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epk: evaluation design 



INTRODUCTION 

In 1970 The Ohio State UnlTOrsity Evaluation Center was funded by the Office 
^TpT^nl?? *°,f®s*f^' operationalize and Implenient a Model Training Project 
(MTP) n Educational EvaluaUon. Of central importance to the project was the 
generation of high quality, transportable instmcUonal systems. These instruct 
k^oal systems ^re the responsibility of the Evaluation Center's Instn^S 
^v^opment l^ Unit. This paper presents the ID Unit's design for the^ 
de^opmenta ^st of an instructional system called EPEC, Ev^uatlng the pSea 
Of Mucational Change, u^^'^^ 

The P^par Is oiganlzed according to the foillowing topical outlines : : y 

1 Delineating Information Needs ^ | 

la THE EmLmilON SETTJNG 

Instryctionai DevelopmenrUnlt 
1.1*2 Description of EPEC 

1.2 INFORMATION REQUIREM^ENTS 

1 •2*1 Antecedents to the Evaluation 

1,2,2 The Decision Setting 

1*2 ,3 the Criterion Variables 

1*2,4 The Deelslon Rules 

1.3 EVALUATION POLICIES 
1,3*1 Access to Data Sources 

1.3*2 Access to Evaluative Information 

1*3,3 Evffduation RaspoiislblllUes and Resources 

1,3,4 ReFiortlng Audiences and Schedule 

1.4 EVALUATION ASSUMPTIONS 

: 1,4,1 Sampling ABSumptions 

1,4,2 Treatment Assumptions 

1«4,3 Measurement Assumptions 

1,4,4 Analysis Assumptions 

Obtaining the Infornnation 

2.1 DATA COLLECTION 

2 , 1 » 1 Informa tion Sources 

2 , 1, 2 Instrumentation 

2 ila3 Data CoUoction Design 
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2.2 DAT/* t».'^GANI2ATlON 

2.2.1 The Unit of Organization 

2.2.2 Storage and Retrieval Requirements 
2*3 DATA ANALYSIS 



3. Providing the Information 

3.1 DfflNlTION OF THE REPORT AUDIENCE 

3.2 DESCRIPTION OF THE fiEPORTING MODE 

1. Delineation of Information Requirernents 

£.1 THE EVALUATION SETTING 
1,1,1 Instructional Development Unit 

The Instructional Development (ID) Unit is composed of the Unit Director and 
four Graduate Research Associates . One graduate student seives as manager 
of the team; another serves as evaluator. The Unit Director and remaining 
two students are writers for the team. Figure 1 presents an orgamzational 
chart depicting the ID Unit's relationshi|5 to the MTP, 
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Instructional Development Is currently based on the Center Instructional 

Devslopinent (cm) Model presented in Appendix A. Since the development 
Of EPEC was Initiated before the development of the CID Model / some of - 
the CID steps were not completed before the ID team began step 19 .2/ 
Developmental Test, In particular, the perfonnance objectives were not 
written (step 7,2) and mastery Items were not written (step 7.3) nor validated 
(step 7,6) , EPEC has already undergone two developmental tests and the 
team is embarking on the third to begin IJovember l, 1972 , 

1,1,2 Dg scrlption of EPEC 

This package responds to the need identified by the OS U Model Training 
Project In Educational Evaluation for materials to train educational evaluators 
and evaluation-oriented educational leaders how to evaluate some aspects of 
the educational change process. The need was identified through a context 
evaluation perform.ed by the adjunct professors / regular faculty and senior 
staff of the OS U Educational Development Faculty, 

EPEC is a nine module Instructional system consisting of 30 hours of instruct- 
ional activities. The materials include an Instructor's Manual, participant 
Manuals and one cassette tape. 

A copy of the orientaUon to EPEC , explaining the purposes and describing 
the content. Is included as Appendix B, 

The system has Qiree general purposes i 

—to provide participants an eKpsrience in- process evaluation 
—to provide participants opportunities to try out some process evaluation skills 
"to provide participants opportunities to identify the kinds of decisions ser- 
viced by a process evaluator 

1,2 INF ORMATION REQUIREMENTS 

1,2.1 Antecedents Jo the Evaluation . 

The primary Impetus for this .study is that evaluation is called for in the ODD 
Model (stop 19,2 .5) to provide the m team with information to aid in revision 
of the instructional system. However, there Is a secondaiy antecedent: since ; 
some of the CID Model development steps were not completed, and since con- 
siderable goal drift has been noted (see recorded documentation meeting of • 
October U, 1972) tfi ere Is the possibility that the outcomes of EPEC in its 
current status may not be suttlciently isomorphic with the training goals of the : 
MTP to warrant continued development . It is hoped that this evaluation will 
provide sufficient Information about IPEC outcomes.to determine whether con- 
tlnued revision is needed or warranted, (See minutes oif the ID team meetinq 
of October 25, 1972.) 



1.2,2 The Decision SetMnfr 



The major decision question to be answered by this evaluation is: Do the 
EPEC learning activities need revision? Decisions regarding revision of 
EPEC are made by consensus among the ED team members. Evaluative Inform-' 
atlon Is thus to be directed to the team and preserved as a record for review ' - 
by audiences specified under Evaluation Policies, below. 

Evaluative information should be summarized for the ID team within two days 
of a module tryout. The summary must be appended with sufficient detail 
to permit in-depth analysis when desireH by the team. 

In order to determine the probable expected outcomes of EPEC a member of 
the ID team eKamined all the learning activlUes and listed the objectives that 
were apparently expected of each. Terminal objectives were separated from 
Instrumental objectives and became the target of evaluaUon. Three kinds of 
Information were then seen as needed: ' 

1. the extent to which objectives are met in training, 

2. the extent to which these objectives correspond to MTP training objectives and 

3. the extent to which these objectives were valid and sufficient objectives of 
an iRstructlonal system on evaluating educational change. . 

The present developmental test is to focus on only the first two information 
needs with later evaluation studies focusing on the third. 

To supplement information about obJectivQ attainment, an examination of un- - " 
Intended or un -explicated outcomes is also to be made. 

1.2.3 the- Criterion Varlahlea ' 

The basic decision alternatives Includei 
1, retain a particular activity 
2 , revise a particular activity 
3, add a new activity , 

Criteria for retaining an activity are specl^^ Failure to meet those 

criteria during the test indicates that the m team should consider revlsiiig the 
activity or adding a now on©. 

For formative evaluation purposes , It is useful to examine a learning activity in 
terms of those characteristics or elements that are changeable. 

The following characteristics were Identiaed the developmental-tast of EPEC. 

1. Substantive Content; The Ideas; skills, principles, concepts, etc. that T 
are the instructional subject of the learning activity. , i ; .v 

2 • Purpose: A statement of intention concemlhg the substantivQ cohtent of - 

the learning activity , - -.- '-.y-- - i . ■ 

3 . Ferionnanced-based Objectives: The operaUonalization of the learning 

activity's purpose— statements of what the participant will be able to do to 
demonstrate that the purpose of the learning activity Is accomplished. 
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5. 

6. 
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. ^h® Efians by which the learning activity's substan^ 

tlve content IS presented to the participants, e.g. , a lecture, a role 
pa^r"' observation schedule, a slide-tape show, or a theory 

Materials: The EhMlcal objects or apparatus used In'the activity.- ' the 
role play instructions, slides, tapes or printed theory papers 
J^ ammq Enviro nment Sp eciflcatmns;: Any special instructions concerning 
the nature of the environment that must be established for the learning 
activity. 

7. Entry Behaviors t (where applicable) General entry behaviors are specified 
for the package as a whole, however, some learning activities require 
special entry conditions to be met if the activity is to succeed 

8. Time Allotmentj The amount of time allocated to an activity as specified 
m the activity's materials. 

9. Feedback PrDceri.ir^s; The means of providing infomation concerning 

participant progress to the participant, the Instructor, and the agency 
providing the training, 

msmsm The chronological position of the activity within and across 
modules. , 



It Is obvious that some criteria are more appropriate during the conceptuali- 
zation and formulation stage of development than during the test stage. There- 
fore the criteria are divided Into formulation and test criteria as follows ; ■ 

Formulation Criteria: that set of criteria to be met while constructing the 
learning activities of the instructional system. FoUowing the C ID Model 
these criteria are applied at several different points during the development 
process, e.g.. Step 7 1 Define Perforrnance Objectives or Step 18: Construct 
Prototype . A learning activity of the instructional system is not considered 
comp ete until the Formulation Criteria have been met for each of thf ieamlng 
activity s elements. Meeting these criteria results in a conceptually sOurd 
prototype needing only emplricarvalidation. Formulation Criteria for each of 
A LEARNING ACTIVm'S (LA) ELEMENTS INCLUDE: ■ lor each of . 

■1. Substantives Content of the LA : 
*V,- /B^^'M^^ically 

pu^ose of the module, 
b. must be validated by experts. 

2, Purpose of the LA : , . ■ 

a, must be consistant with the purpose of the module. . 

b. must be accomplished as a necessary condition to • 
meeting the purpose of the module . The conditional 
bases may be logical, empirical, or theoretical . 

^ ' - ■ ■■■■■■■■■■■■■■■■ 9 - ■ ■■■■■■■ ■ V-.-;--.v: 
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3^ Pgrformance-baaed Qbleetlves of thfl T.fl. 

a, must state the learner. 

b, must state the beHavlor product, 

c, must state the cue or stimulus condition, 

d, must state the limits of acceptable response. 

e, must be logically related to the purpose of the 

LA so that if the objective is attained, it can 

be inferred that the purpose of the 14 Is partially 
accomplished, , 

*s a set, sufficiently cover th© substantive 
content of the activity, 

g. must take into account the limitations of reality. 

h. must be necessary to the LA purpose on logical 
or theoretical grounds. 

4 . Media and Methods of the lA ; . 

a. must be appropriate for . 

1) the content, 

2) the target aydience, ; 

b. must be consiitant with 

1) learning theory, 

2) communication (a/v) theory, 

c. must be sufficient on logical or theoretical 
grounds to accomplish the LA purpose. 

•d, must be sequenced in a conslstant and logical 
manner, 

e. must be least costly and elaborate for the job, 

>. Materials of th» T.A. 

a. must be editorially adequate, 

' * Learnlno Environment Speeiflcatlons of the LA ; 

a, must be necessary and sufficient to accomplishing 
the activity' s purpose on logical or theoretical 
grounds, 

. Entry Behaviors of the LA ; ' " 

a. must be necessary and sufficient to accomplishing 
the activity's purpose on logical or theoretical 
grounds . 

. Time All otment o f ths T.A; - ; 

a. must be appropriate for the relative value of the 
activity. . 

. Feedbac k Procedures a f thm T.fl.' ■ - 

■ a.- must be conslstant with learning theory. -." . 

b. must explicate a remediation procedure v/hen 
applicable. . i r\ . 



10. Placement of the LA ; • 
a. must be conslstant with a model for the system. 
ID. niust be conslstant with learning theory. 

^ Criteria: that set of criteria which must be met during a developmental 
tT^^^^'^"^^^' 'hat does not meet these crlteS^t 

t^^^^^r^' "^^^^^^ revlsed,the original formulation criteria 

1* Substan tive cont ent of thf^ T.A* 

a. must not ba provided by the Instructional manager - 
should be perceived by participants as relevam 
2, Pufposa of fhP T Zk* (none) 

^ • Perfonrtance-bas^d oblectivg^s of the T^f - 
a* must be attained* 

4, Media and M#^f-hods of th^ 

a, must be perceived as satisfactory by 

1) participants^ 

2) the instructional manager. 

b* ^^st be Implemented as specified* 

S* Materials of the 

a. must be perceived by participants as 

1) ledgible 

2) convenient to use 

3) not overly expensive 

4) appealing ^engaging 

5) understandable 

b* must be perceived by the Instructional manager 
as appropriate ^ 

G/ must be durable enough to fulf^ 

Learning Environmg^nt ^p^r-f fixations of t^^ T/^v 

a. must be perceived as implemented by the obser^^^ 

b, must be perceived as necessary and sufficient by 

1) the observer, ' 

2) the instructor^ and 

3) the participants, 

7* Entry Be haviors of thn T.A^ 

a. must be perceived as necessary, sufficient and met by 

1) tlie obsen/er, and 

2) the jtastructlonal manager. 



b, may be objectively assessed when needed. 

8» Time allo tment of th^ 

a« must con^espond with theractual tl^^^ 

dm^lng the developmental test, 
b* must be perceived as appropriate by 

1) participants^ and 

2) instructional manager 



9* Feedback PronRriures of tha lAt 

a* must be perceived by parUclpants as adequate 

and not disruptive. , 

10* Placement of th^ LA** ^ 

a. must be perceived by^partlclpant as appropriate- 
in temis of sequence arid continu^^ 

In addition to Infomiatlon about specific learning activities it is also desirabl 
to obtain some macro information about the system as a whole. Two addltiona 
test catena are intended to provide this macro informations 

11. The outcomes of EPEC must be isomorphic with the training ' 
goals of the MTP as perceived by ' t . . 
a, participants ^ / 

b» EPEC develop'ers 

12. The participant perceived outcomes of EPEC must 

correspond to the outcomes intended by the develop-^ 
ment team* - 

since all Information needs can not be anticipated In this design, a summary 
evaluation activity to foUow the last module Is planned. The evaluator 
is to keep a log of important Issues and questions that occui" and remain un- 
answered during the test. Then the summary evaluation is to Include a parti- 
cipant interviow and other activities to provide the needed Information, 

Since this evaluation relates to a developmental test/ ONLY THE TFST CRT- 
TEm ARE TO PUY A ROLE IN THE EVALUATION DESIGN . ' In Table T each of 
.ne Test Criteria Is presented, followed by the criterion decision rule, source 
of the relevant data and name of the Instrument deslgried to obtain the data, . 

1.2.4 The Decision Rules ' ' " 

If decision alternatives are to be weighed in light of specified criteria , the 
nature of the relationship between decision altemaUves and criteria must 
be clear. This relationship may be made clear through the establishment " 
of decision rules that describe what circumstances dictate the adoption of. ? 
.what decision alternatives . Ducision rules are not dlffidltLlt "to "determine ' ' i i 
when the declslori Involves choice among alternative strategies to a given;-—', 7'' 



Matrix of Information "Requlrenient^ 



Critena 



Decision Ruj e 



Datajource Instfument-':-" 



L" Substantive (content ofth^ W . 
a. must not be provided by the instructional manager 
- ' 'b. should be perceived by participants as relevant 
2 . Purpose of the lA : (none) 
'3. Performance-hasRd Objectives nf th^ ]!■ 
a. must be attained 

4. Media and Methods of the LA ; 

a. must be perceived as satisfactory by 

1. participants, '' 

2. the instructional manager, 

b. must be implemented as specified. 

5. Materials of the U - 

a. must be perceived by participants as acceptable* 



f provided content=0 ' 
positive riSponse285% 



Observer 
Participant 



Observation Sche 
PROBE 



Fir Cent (Ki90%)>90| Participant FerfomancaTe'st; 



affirmative response285/ 
affirmative reiponse 
f deviations-O 



Participants 
Inst, Mgr. 
Observer" 



^. ,.....v.u uy pdmcipantsas acceptabli* affirmative response>e5% Partirlnflnt 
b. must be perceived by the instfuctional manager ^ ^ 



as appropriate 
■ e. must be durable enough to fulfill their function 
' kaLninq Environment .Spprifif^ati^n'^ ^f thf- Ih: 

a. must be perceived as implemented by the observer 
. b. must be perceived as necessary and sufficient by 
' .1. the observer, 
. 2. the Instructor, and 
3. the participants. 
- 7. Entry Behaviors of the LA : 
, a. must^be perceived as necessary, sufficient and met 

1. 'the observer, and 

2. the instructional manager" ' 



*See test crltirla for a breakdown of this category. 



affifmativi reeponsg 
affirmative reiponse>e5% 

f departures=0 

f reported problems=0 
affirmative response 
affirmative response^SSI 



f reported problem=0 
affifmitlve.fesponse 



Inst. Mgr. 
Participant 



^probe;"^": 

Instructor PROBE'.! 
Obien/atlon Schii 

PROBE ■ " 

Instructor PROBE I 
PROBE ■■■ 



Observer Observation Sched 



Observer 
Inst, Mgr. 
Participant 



•Observer 
.Inst. Mgr. 



Observation Sched 
Instructor PROBE'^ 
PROBE " ■ -■:,! 



Observation Sched 
Instructor PROBER; 



^ . ..... , '■ ,. ; Criteria . . 

- 8. Time allotment of the LA! - 

• a.vmuat corris pond with the ictual time consumed 
; ^ b. must be perceivid as appropriite by - ■ 
I. participints, and 
.2, instfuctional manager 
; 3 . Feedback FrQcedurei of the LA: 
" "a. must be perceived by participants as adaquate 
■ and riot disruptive' 
40. - PlacjmGnt of the LA :. 

a. must be perceived by participants as ippropriate 

U. The outcomoi of EPEC must be iiomorphic with the 
: " training goals of the MTP as perceived by 
■ *a, participants • 

b, EPEC developers 

12. The participant perceived outcomes of EPEC must 
"■■.correspond to. the outcomei Intended by the develop- 
ment team 
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f- discrepanciei-0 



'Data- Source—'- In|tnjmM'-% 
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Observer Observation ScHec 



PROBE ■ J: 
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affirmative response 85% Participant PROBE 



affirmative responii 851 Participant • Module Questionnc 
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undetermined 
undetirminid : 



undetemined 



Participant SAES 
Developers SAES 



Participants & Final Questionnalri 
DevelODers ■ 



. . ■ • - ; ,._ ii/i::: 

end. Once the reievant.crlterla have been Identified, that strategy best- 
meeting those cmeria-ls t^^ However, in the inetmctianal de^^^^^ 

ment process a different kind of decision setting often presents Itself The " 

" ■■such'rnr"''.f ' f T-^" ^^^^'^"^"S or revising an ll baled on 

.,s^ch infonnatlon as the LA's effectiveness in meeting objectives Wits- use^-- " " 
falnesa as perceived participants . But, how weTl Should objectives be ' ^ ' ■ 
met? How useful should training be percelved-b. what proportion of 

crmcaTHmit; ,f ^f-f arbltrafil/eSoTfying th. 

critical limits in a decision rule? ^ • 

"tlT^L^f^^^^^ ^^""^ proposed for the Instructional development 

dec s on to revise/retain an LA. An often used convention Is the 90-9? 

lerit qT/nf'th° " 3°^^- °^ participants meet at - - - 

least 90 /. of the objectives . This decision rule is. probably an operational- 
l^atlon of "most of the students should learn most of the material .'' llimllar 
. rule was proposed ty FC Butler (1972) involving use of the standa;d deviation ■ 
of the nonnal curve. After de. .loping a sufficiently validated test 85% of^ "" - 
trained population should meet 100% of the objecWves. 

Decision rules such as these, stated strictly In terms of objective attalnmint " 
may be practical In relatively few situations. It is difficult to take such a f ^ 
tltt^nl "^^P^^ly ^hen repeated eKperlence has shown the difficulty Ld ^ 

must ask himself if he seriously Intends to revise for mastery . If other varlkbles 
enter Into the decision, they should be included In the decision rule! Se 

:S:taThisf ::L'ebt:! '^^^"^"^^ " " ^""-^^ - 

These dec2Slon^rules_^can also force the decision maker Into a defensive pos- 
ahil' rl l f^oin the mle to those to whom he is account- " 

• Z^^ decision rule must be -Ve-stated^ch-time„ there is a departnm ■ 
-and consequentlrthe^^aiBrloses^its-value in providing; for systematic con ilSSion^ 
and selection of decision alternatives. ■ ^^SESiloa. 

In order to avoid these problems and^yet still .avoid arbitraUon in stating cnticai - - 
limits it was decided that critical limits of the criterion variables should - 
relate to the relative importance of the variable and secondly, that the decision'' 
rules shoula dictate not "revise" alternatives, but only "consider revision" 
alternatives . ^evidence suggested a need for revision as dictated by the 
d^c J nn ther set of practical criteria would enter In ^o>the final 

decision. The effort to systematically determine and relate these practical ^ 
criteria to the decision altemaUves by means of decision rules was considered 
too great. These practical criteria may include! 

1, Strength of the evidence suggesting revision. 

2. Seriousness of the consequences of identified deficiencies, 

3, Cost In time, effort and money to co^ect the system, 

4. The m team's confidence in the prognosis for the system's ailments, j 



During actual implementation of this phase of the evaluation design there 
. is to be an examination of the kinds of variables that do enter into ID team 
discussions and the decisions made to determine the feajlbllltK of system- 
atically gathering data about these variables, ' 

Decision Rules for affimlng that criteria are met are listed in Table 1, ' 
■ Criteria that are met lea^ to the decision to retain a learning activity in its 
present form. Unmet criteria lead to the decision to consider revision. 

In general, the critical limits for decision rules were set as follows: ' 

For criteria that relate to participant attitudes toward training, the c-ltlcal 
limits were set at 85% of the test population. - 

The 90-90 decision rule convention was adopted for attainment jf objectives 
Since the mle would only leaH to a consideration of revision the developers 
.„could afford to majce an error on the. side of too strict a rule rather than pass ' 
over package weaknesses-that might be fairly easy to remedy ' • 

Criteria related to the correspondence between Intended and actual procedures 
and time lines received zero frequency of "deviations critical limits, | , 

When considering participant feedback and recommendations it Is reali^.ed that 
It is the quality of the remark, and not the number of people who ccnfin- it only 
that should be considered. Therefore it is necessary to prevent the decision " 
rules from filtenng out important information from the decision makers— a's 
might happen If only those data not meeting "retain" criteria -are displayed. 
"po accomplish this, all open-ended remarks are summarized In the information 
displays for the decision makers . 

1,3 EVALUATION POLICIES^ 
1,3,1 Access to Data Sources 

^%^^f''^'^''^ restrictions to data sources are foreseen. Generally, data sources 
lOi^de only the participants, instructors and observers of the developmental 

1,3^2 Access to Evaluatlva Information . - 

However, access to evaluative Infonnatlon will be restricted to the following 
personnels - 

1^ AEslgned staff from the Evaluation Servlcas Component 
■:: 2 V The MTP Director 

3> USOE and NIE personnel assigned to monitor the MTP ■ - 

4 . The ID team 

A final summary report of findings U to be nade available to the public. 



■ ? ^^^^ instructors and participants have a potential in- 

fluence on revision aeoislons and might be considered an audience for 

th"s"u «of?t"r,L "r^^"' P^"-iP"'s and instructor are 

1-3.3 Evaluat ion Rpsponsibili ty ^-oH n^-^^,,j.rri 

pecte. to exceea 10/. of Lhs resources allocated to Instructional Development 

In order to avoid problems of co=optation and role conflict, It Js understood 
that the evalutor-s responsibility is limited to evaluatlon^'that a ha does 
not engage m qevelopment of learning activities. - 

^'•3.4 Reporting Audiencos and Snhcdnip 



The ID team is provided with evaluative Information within two days of each ' 
recom f:''""^.^'^'^' ' A final report summarizing flndlngs'and e" h ' 
recommendation ,s presented within 30 days following the ninth m^ule 

The evaluation design may be changed only by consensus of the ID team after 
examination of some of the evaluative data generated by the design. 

1. 4 EVALUATION ASSUMPTIONS - 

1.4.1 Sjampllng Assumntions / 

The targ includes evaluation -oriented educators m 

-n^rK ^'"^^^ °' '"'^^'^^^ However the evaluator does 

possm'^1^' '° ''h'""'''^ °^ '^^Set population nci^" 

fhLi^ very adequate noundarles defining all possible members . 

There are no restrictions on the population except that college level !S nlng 
Wt^W^PT ' '7 developmental test the sample wilfS^^^n^ply 
of^the Eix= graduate students who enrc. led for EPEC Fall Quarter 1972 The 

in the design but yields ^aktio^^ ^n^^^l^J^^^^ 

are examined for their Implications , the sample size should be kept In'mind. 



con^ 



1,4 . 2 Treat m en t Assumptions 

,,..This design attempts to establish neither internal nor externaL validity of 
the treatment outcomes. The final report can not, therefore, -attest with 

-^^'^"^^^ that the outcomes are. due to EPEC or are representative of outco 
that might be attained by other members of the populatLn: 'Th s L a 
siaerable weakness in the design and hopefully EPEC.outcomes will be 
vaUdated In further tests such as the Field Test. ^ This deve^opmTn al test is " 
still expected to yield important lnformatlon--whather .ot not the lle^^^^^^^ 

— -e^wientiSphic 

r.4>3 Measurement Agsum pt-innQ - 

There are three general classes of phenomena to be measured in this test ' The 

first is participant and instructor affective reactions to specified variables 

of the learning activities . Essentially, it must be determined whether parti- ' 

- ist'sn .h' f ' '° satisfactory along, the dimension specified 

.. IS assumed that participants' responses to a scale of "satlsfactorvness " wll 
adequately reflect their affective reactions " atistactoryness wil 

The second class of phenomena to be measured is the physical occurrence of 
specified events: trainer behaviors and procedural or temporal drscrepancies 
It is.assumed that the frequency of those phenomena can best be measured by 
an observer using an observation schedule. measurea ay 

Iplc^sV^nrf P'^^"^'"^.^^ ^"^^"^^ participant performance In relation to 

that thele n^""?-^ '""''"^l- '"^^P" '"^^^ questionable assumption 
IS that these objectives are Indeed a valid representation of the universe of 
process evaluation skills for change processes. Criterion referenced^ tests 
are used to measure objective attainment so the validity of-the ^s j^s Mnge ,n 
part part on the validity of the inferences from correct responses to skill 
possess on Yet, do to a lack of time and resources, these inferences are 
not carefully examined and tested. ietences are 

Th^e are to be no formal tests of instrume^ or objectivity except 

>hat performance tests will ba scored independeatly by each of the D team Lmh 
and the results analysed for agremment among scorers' Ihe reporting p'am^te's' 
for hi. ana ysls will be a simple percent of so o ring oases (across items'nl 
participants) In which there is perfect agreement among the five scores and the ' 
percent of cases in which there is more than one dissenting score A mire 
sophisticated analysis is not warranted. The expected value for he percent of ^ 
agreements is greater than 84%; for the Percent of di sag re em entire ss^^^^^^^^ 



1.4.4 _Analvgls &Q«Mmp^,-„n^ """^^-^-^ ^ 

In general the Information gathered ff^r . " 

or absolute standard. The standards ar/^'^' ^" °^ '""^'^ ^"'^ ^ "^^^tery ■ 
reporting parameters for the' e com' ""-^i • The- ^ 

These analyses require no sneoTcT.f f f^-QUencles and percentages . 
•~ outcome variablesVcross aeUv'^esTnd ^-umptions. ■ Comparl sin. of ' 

of respondents meeting erlter on l"itf ^'f , °f 'he mean percent 

statistics are expected. ^° analyses requiring inferential 

2. Gbtalning the Informatlon.^^^^^^-^^^^^^^^ : : - 

2.1 DATA COLLECTION _ ' v 

2 . 1 . 1 Informat ion Son rr-oc / 

The sources of, the Information required for earh , " 

presented in Table 1. The samoi^ i^^f 7 I evaluation criteria are' 

EPEC for fall quarter plus° wo in frS^^^ 

the other for the remaining five Soml nf ""'^ four modules and 

an:observer. The samp'l ^ tS te^'^^jr 

of the target population of partlcTnant'' fn.^ "f^ecessarily representative' 
Pllolt inferences to these P^ auS^'Vh^ °' ^ 

atlon about the sufficiency orEPEC il ^^^^^^^^ the least, infonr,^- 

specified population. ° some members of the ' 

2,1.2 Instrumentatj nn v # 

The Instruments to be nq^H i^n fk^ ^ ^ 

foUows, wUh . Ust ofThese instrutems;" ''^'''^ Appendix D. I.Ue 2 



XABLE 2 
Instruments 

Instructor PROBE 
Module Questionnaire 
Observation Schedule 
Perfonnance Tests 

Final Questionnaire 

Tape Recording of Sessions 

Interview 



Figure 2 portrays a .general model of the evaluation design for a given module. • 

All instruments are listed together with the individual who is to respond to the 
instrument and the administrator of the instrument. It Is the observer's respons- 
-ability to gather, organize and return the data gathered to the evarutdr. 

2.2 DATA ORGANIZATION 

After a period of initial trial much of the data is to be stored and analyzed by 
computer. All data cards contain a common field including columns 1 through 10 
storing the data coded in Table 3. This table also allocates card numbers to 
the various instruments. • Instrument codes and column numbers are printed 
directly on the instruments . }' - : - 

2.2.1 The Un it of. Organization 

Subject by subject responses are retained as the unit of disaggregation. 

2.2.2 Storaqe^nd Retrieval Reciuirements 

The data is stored in raw form in a large loose-leaf binder under the care of the 
evaluator v/hile analysis is underway. After the test, it is to be kept inacabinet in 
the Resource^pata Bank. Storage of data cards will be specified after tne ' 
decision to begin use of computer data processing . The instruments and 
analysis forms are kept by the ID secretary. _ 

2 .3 DATA ANALYSIS 



This section presents a strategy for obtaining needed infoirnation about EPEC 
revision from data collected in a module tryout. : . - J 

As Indicated above ^ various criteria must be met by each LA if the LA. is to ' 
^tand unrevlsed. Specific decision rules Indicate when the criteria are to be - 
cohsidefed as satisfactorily met. After data are obtained- the remaining task 
is to organize it in^a way that shows whether these criteria have been met . To * 
facilitate organization of the data a series of analysis forms are provided in 
Appendix E . Table 4 lists these analysis forms and shows their relation to the 
instruments and information sources presented above. 

A brief overview of the analysis strategy is presented below. A more detailed 
°' analysis instfuctions Is provided in Appendix F .' 

The analysis procedure is basically a=two stage process. The first stage Involves 
completlngJ heAnalysis forms which organize the data for the second stage. 
Tne second stage is an iterative process involving several steps. First, th'e 



hqml ■ 
of iviluitton Aetivftlis 





Instrument 

1 - Bickground Information 
2-- PROBE General Instructions 
.3 - Observition Schedule 
^ ' Begin Tape Recording 
5 = PROBE 

S - Instructor PROBE 

7 = .Hastery Test 

8 s Hodule Questionnaire 
9^ Final Questionnaire 



Administrator Recipiint 

Instructor : Participant & Instructor 

Instructor Participant 

Observer Hodule 

Obsurver Hodule 

Self Pirticipanf - 

Self '^Instructor 

Instructor Participant 

Instructor Participant 

Instructor Participant 
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Com mon Field 
Column 

01 - 02 

03 - 04 

OS 



06 

07 - - 
08-09 
10 



- - TABLE 3 

CODE SHEET 

Variable and HnriW" Paf:^-^:; .1 
Instructional System Number 
Test Site Sequence Number 
Instrument Number 

0 - Background Information Questionnaire 

1 ^ PROBE 

2 - Instructor PROBE 

3 = Performance Test " \ 

4 ^ Module Questionnaire' 

5 = Observational Schedule 

6 = Final Questionnaire 
Module: Number 

Group JJumber. _ ^ - / 

Partlclpant-ID. Number 
Open ' 
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Information Provision^ 
for 

Module Report 



Mnformation Source 



Instrument, 



Participant 



Instructor 
Obseivar 

Evaluator 



PROBE 



Module Questionnaire 



Performa^rice Test 



In s true tor Probe 



Observation Schedule 



4 



Analysis FomiS; "/^^ 
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Table A- 2 PROBE Results by Activity^ 
Table A-1 PROBE Summary v-tr- ^k '^1 
DisplayA-1 PROBE Open Ended kcmark 

TableA-3 Questionnaire Results ''.'V-' 
Display A-1 Questionnaire Open 
Ended Remarks 

Table A- 4 Performance Test Results'^:! 
Table A- 5 Performance Test Fjlequeney 
Distribution | " 

Table B-1 Instructor Probe Ratings 
Display B-1 Instructor Comments 

-. - . - , , . » . " •. . • 

Table C-2 Agenda Item Data " ' v 

Table C-1 Activity Data 

Display C-1 Observation Comments 

Table»l Module Summary - - _ 



criterion variable^ of Interest to a particular element of an LA is identified. An:'i 
example might be "performance-based- objectives of the LA must be attainea^-^S 
. Second, data relevant to this criterion yariable are located on the analysisx'/-t-i 
forms that were prepared In the first stage. Third, the results obtained for thei 
criterion variable are noted in terms of the predetemiined decision rule. If the""-^ 
-criteria are met, the next criterion variable is considered. However, if criteria- 
are not met, this result is noted in Table 1, the Module Summary. Entries in 
.-thls table indicate that.a particular element of the activity is not satisfactory : .:. 
and, _thrqugh,a;codlng. system, refer. the reader tosupporting information in the 
..appendix of the report. Whenever an entry. is made in Table 1, indicating a 
, need for revision, a discussion of the results relating to the criterion variable 
under, examination is developed in the body of the Module Report. This dls- 
cussion considers all tlie information about tJie activify that-is^avaiiabie, 
. presents both the evidence suggesting revision and -arguments for retaining ' 
the activity, offers explanations about the nature, of the problem and recommends 
-what action might be taken to improve the activity. The whole procedure is 
• then repeated for the next criterion variable until all the elements of all the 
activities have been examined,' «. ^ . -'. .- :~ V'':"'r. 
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3, Providing the Information 

3 . 1 Da'INITION OF THE REPORT AUDIENCE 

The report audiencQ for most of the evaluatlvo Information will include only 
the ID tGam . The Final Report ia made civalloale to all MTP componont 
^ir?.^L°*''Sr NIE monitors, an 

3.2 DESCRIPTION OF THE REPORTJNG MODE 

The reports to the ID te^im consi^L of the completed analysis forms presented 
after each modulo of EPEC . Speipial staff meetings are gs tablished for the 
purposQ of roviewing the informQtion. 

The rinal Report is a printed paper condensing the major findings and con-^ 
elusions of the evaluation studyv 
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The Center Instructional Development Model 
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I N STR U CTl 0 N AL D E VE LO P M E NT AT 



-\-- 



THE OHIO STATE UNIVERSITY 



EVALUATION CENTER 



The Ohio State Universrty Evaluation Center lias evolved a 
systematic process for the development of educational product^. 
This document describes this process/ both to identify its concepts 
for Center personnel and to provide information for persons not 
iated with the Center, 



August, 1972 



Jack Sanders, Director 
Instructional Materials Development 



Paul Carlson, Research Associate ; Jerry Adams, Research Associate 



Kay Rofkahr, Media Specialist 
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intfoductlon \ ^- 

* ■ * ■ . . , . 

. - The Center Instructional Developme^^ri^^^^^^ 
is not a new discovery. The CiD Model has evolved from the many instruc- 
tional development tasks perlormsd by The Ohio State University Evaluation 
Center since its inception in 1965. Tasks, timelines, personnel, and pro- 
ducts of the Center have varied over that seven years . But the dedication 
of the Center to the improvement of graduate education, generally, and the 
training of educational evaluators, specifically, has remained constant. 
The CID Model reflects the Center's dedication to instructional excellencei 
as well as its past experiences in development. It builds on this dedica= I 
tlon and experience, presenting a systematic process to guide the Center's 
future development efforts. Specifically, the CID Model is Intended: ' 
a) to m^re efficiently fulfill the instructional development prescriptions 
of the Bargar "Model of Graduate Training" and the Bunda-Stufflebeam 
"Universe of Evaluation Competencies" substantive model; b) to delineate 
the processes followed by the Center in developing instructional products; 
c) to provide a system for Internally and externally evaluating instructional 
development activities of the Center; d) to facilitate cSoperative ventures 
with other developers and their^agencies; e) to provide for students see 
evaluation/devolopment assistantships an o very lew of the type of activities 
the Center's instructional development unit performs. 



The CID Model is depicted in four phases: context phuse, Input phase 
process phase, and product phase. Each of these phases are charted out 
in -rms of task=flow and decision points. Next, task-flow is explicated 
along with appropriate criteria for decision points. Then, the entire array ; 
of CID Model processes is laid out for inspection. Finally, a design for 
validating the model is presented. 

This model may Of may not be linear. Under ideal conditions , where 
developers ere not hampered by financial constraints, unrealistic timelines 
personnel problems, and antiquated production facilities, the model may ' 



be consistantly followed in a linear fajhion. But the Center has found that, 
even under "Ideal conditions," strict linear use of the modi! is sometimes i 
not feasible. For example, some Center work'lnvolves adapting or adopting 
-produc.ts==pfGduGed:elsewhere. Naturally, the latter do not follow the process 
from beginning to end , but enter the process at an appropriate stage along the 
way. Additionally, some Center products may enter the model at more than 
one point simultaneously, e.g. a development team may be chosen before 
the product is through the context phase. Nevertheless , the model and 
processes presented here do reflect the standard course of development 
taken by most instructional products of the Center. 

The CID Model is presented here as a prototype. It has been validated 
by external judges as conceptually adequate, but has not been through a ^ 
complete pilot testing at the Center, nor has field testing estafaUshed its 
generalizabUity. It is presented here as concrete evidence of tTe Center's 
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dedication to institutionalize a systematic approach to the development of 
instructional products. 

Figure 1, following, charts the Context Phase of the CID Model. 




Identify needs, 
opportunities and 
problems (NOP) 



o 



Major; 
deGision 
point 

Feedback loop 
for process 
or product 
rafinement 




Identify specific 
goal not being 
attained 

4. 



Write problem statement 
(program goals) 



Search for and catalog 
existing instructionarmat- 
erials related to problem 
statement & define other 
relevant resources 

i — - 



Document alternative 
solutions for achlevin 
program goals 



1 



Identify best solution 




Proceed 
to 

Input Phase 



Flgure-^ir CONTEXT PHASE 
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Context Phase " 

The Context Phase of the CID Model is concerned with deaneatlng 
the/evaluation training products that are needed, the unique op fiortunitles 
which come to the Center for developing products, and the problems which 
'prevent the Center from fulfilling identified needs or taking advantage of 
extant opportunities. The objective of the Context Phase is to prepare for 
the appropriate decision maker a report advocating start-up on development 
of the most needed Instfuctional products. 

Criteria for making decisions depicted in the Context Phase are: 

1. That there is empirical support documenting the existence of 
specific needs , 

2. That development opportunities available to the Center speak 
to documented needs . 

3. That logical argument indicates there are no anticipated, insur- 
mountabla Internal or external problems that would confront the 
^®^®^°P6fs in meeting the documented need. 

4. That the proposed program's goals are not in conflict with the 
goals and subgoals of the Center. 

5. That justification for the proposed "best solution" is empirically 
• ^ based. 

. 6. That estimated cost-per-product is Justified in terms of Instructlona 

need being served and number of potential users . . 
Figure 2, following, charts the Input Phase of the CID Model. 



Major 

decision 

point 



Feedback 
loop for 
process 
or 

product 
refinement 
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Define instructional setting 



Define performance objectives 
In relation to program goals 



Identify key concepts ^ scope 
and sequence for entire 
instructlonar package 



^ ^ = 



Break key concepts into 
modules 



^ — - - - -, 



Identify scope and sequence 
of each module and match 
each performance objective to 
appropriate module * 



1 



Specify appropriate = 

instructional strategy 



_ _ _ _ J 



Identify resources needed 
(development team , consul-^ 
tants, specialists, materials, 
equipment) 



Estimate required time and 
schedule activities 



Make cost estimate 



I 



ERICl 



Write summary of context 
. . and^riput^ phases 
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Input Phase 

The Input Phase of the CID Model is concem^^ : 
design of an instructional product which Is to fulfill a high priority training 
need. The objective of the Input Phase Is to prapare for the approval of 



thadevelopment team the design of a specific product and a detailed summary 
of resources necessary to implement the design. ^ J-."^- 

Crlteria for making decisions depicted In the Input Phase aref 

1. That performance objectives and key concepts are as specific 
and clear as possible. 

2. That key concepts are broken down into logically consistent 

modules, , 

3. That each modulo contains logically constant perfornjance pyectives 
,y 4. That the selected instructional strategy is consistent wltK t^ 

performance objectives. 

5. That needed resources are available to development team . " 

6. That suggested time Schedule can be adhered to by the de^;elopment ' 
team. " ' ■ ' . \ ' ■ 

1, ^..lig^r? 3^ Mlowing cha rts the. Process. Phase of the CID Model v - 



i 



Approval 
given for 
proceeding to process 
phase 
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Major 

decision 

point 

Faedback 
loop for 
process or 
product 
refinement 



Choose development team / 
prMure needed materials 

and "equlpment"'^^^''^^:"':^'" ^^^^^^^^ 



Generate organisational 
plan for development 
team 

— — — — ^r~~— -~ 



Construct prototype 
I 



I 



Feedback to : : 
Key Concepts 
or Scope and 
Sequence 



Administer tutorial test 



Make revisions as needed 



D e s ig n ^proc e a s evaluation 
and instrumentation for 
developmental test 





Identify 8 


^10 from target 


audience 


within commdn 


locale 





Administer components of 
prototype (developmental 
test) 



Make minor revisions 
as needed 



T 



Make major revisions as 
indicated by evaluation 
findings 



T 





erJ:SS 



Development 
team decides if 
rototype_is adaqua^^ 
for field ^^.^^^^ v 
Proceed ^-'^^test^^^ Recycla 
to— 

Product Phase : . 
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Figure 4; PRODUCT PHASE 



Process Phasfl 

The Process Phase of the CID Model is concerned with building andV 
P*°'°'yP?iHt£UCtional pro 
is to develop and refine the product so that it is satlsfactofy for field testln 
/ : Criteria for making the decision that a product Is ready for field test 



are: 



1. Perforrrince data indicates that product meets its performance 
objectives . 

2. Adequacy of pv^ uutlon d.'^^ign, instrumentation, Implemontatlon, 
and repc: I , 

3. Adequa^v of acgumentatlon of pilot test ■ 

4. Extent of necessary revisions after last devGlopmentartest: 

5. Appropriate sample t-om target audience was used for field test. 

Figure 4 , following, charts the Product Phase of the CID Modet. . 



•.:cgi.^. 
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-n.' 'ivr ' 



TV f 




Contact dlsseminatrng^ ^ ~ " 
agency m their Information 
needs 



T 



Identify field test sites and 
negotiate necessary 
arrangements 



T 



Design product evaluation 
and instfumentatlon 



Major 

decision 

point 



Feedback 
loop for 
process or 
product 
refinement 



Identify and train 
Instructors 



T 



Administer field test 




Feedback to 

Developmental 

Test 

! 



Prepare field test report for 
dissemination with the 
instructional package 



Disseminate appropfriata 

number of compieted packages 

to appropriate agencies and 
Duhlisher ■ : 
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- ■ Product Phasp ■': : ■ 

The Product Phase of the CID Model Is concerned with the obj^^^ 
, -field testing (summatlve evaluation) of an instructional product; I The ■ 
: objective of the Product Phasejsjo empirlcally esta^^ 
-^; tionai ::produot:does-^^ and hence is J 

ready for dissemination. 

■ Criteria for making the decision that development of a product has 
been successfully completed are: 

1. The benefits of applying the product outweigh the costs of 
adopting It, _ 

2. External evaluator's findings certify objectivlty'of field test. 

3. Field test data indicates that performance objectives were 
attained, 

. 4. Field test data indicates that instructor training was adequate. 

5. Field test data indicates positive affective response of instructors 
' P«'"^iclpants to the Instructional product. " ' 

6. Field test data indicates the product is politically and morally 
viable for use of target audience. 

- 7. Field test findings indicate the product (with minor revisions) 
V is ready for dissemination. 
Thus^far, this document has considered the major activities declsloh 
points, >and-crlteria of the CID Modei. Subsequently, a complete delineation 
:of major^and minor tasks associated with the Center's Instructional develop'- 
ment is presented. • ■ 40 - - ■ 



Define Need/Opportunities/Problems (NOP) ~ ' 

-1.1 Define source of OP - ' ■ - 

'1-1,1 ^ Students in evaluation training 

1.1.2 Curriculum Task Force ^ . : . . ' 

1»1.3 Universe Data ' / . ' = ' 

1-1.4 Training needs-analysis of Consortium' . 

' -1.1-5 Associated OSU Faculty, Dept. 7 or Center 

1.1.6 RFP 

l*l-7 Oregon Data 

1^.1.8 Other; 

1.2 Write NOP as problem statement (program goals) 

1.2.1 , , Identify specific goal not being, attained . ! 

1.3 Validate problem statement with other sources \ 

1-3,1 Students in Evaluatibn-training (principal source 

for validation) '7 = „ " 
1-3-2 affiliated evaluation agencies ' 

1-3. ,3 Universe Data* ^ . 

1*3,4 Othef Jabs. and. centers " 

_ 1.4 Begin log of development effort ^ ^ 

Define Available Resources . - 

2.1 Search for and catalog existing instructional materiais'related 
to problem statement ^ - . 







2.1.1 


PRDB 






2.1.2 


' - NCERD "Hot Line" 








CL'DaR 






2.1.4 ' 


TAP 






2.1.5 


DID-AEi 'contacts ■ 






2.1.6 


Other 


2 


.2 


Content experts . 






2.2.1 


l.n-house 






. 2.2.2 


other 


2 


.3 


Instructional Developers 






2.3.1 


in-house — 






2.3.2 


other 


2. 


.4 


Evaluators 








2.4.1 


In-house 






2.4.2 


other 


2. 


S 


■Consultants, 


i.e. , media specialists - 






2.S.1 


in-house 






2.5,2 


other ■ 


2 . 


6 


Finances 








2.6.1 


from existing development funds 






2 .6.2 


from new funds 






2.6.3 


. _from other sources -,- .- . 



5?^^- 2.7 Production facilities : - ^ " 

=- 2.8 Probable sites for tutorial , developmental and fleld tests ' ' 

- ; 3 .0 Pocument Alternative SolutionB for Achieving" Program Goais . ^ I 

4.0 Identify Best Solution - - ■ ' ' - . . . 

S.O Prepare. Context Report for Decision Maker . 

6.0' Define Instructional Setting 

6.1 Target Audience " ' 

6-1.1 Graduate Students 

6.1.2 Workihg evaluators ' " : 

6 .1,3 Beginning paraprofessionals - • - • 

6.1.4 ; Others . ' ' 

6.2 Physical facilities for target audience - " 

. : 6.2.1 Space ; _ ■ • 

6,2,2 Hardwari / 

' , . 6,2.3 - Software . ^ . 

' 6.2*4 Libra^' - 

6*2 .S Other 

6 .3 Instructors, if appropriate . , ^ 

6 .4 Community resources; \ - ■ 

.7,0 Deflne^^Performance Objectives In Relation to Program Goals ^ - 

7.1 iResearch existing banks of objectives 
^ :; 7.2 Write finalized performance objectives . ^ ^ 

^^7 ,3 Write mastery items . ■ 

. ~ 7.4- . Write needed, entry behaviors ^ ' ■ . . 

. 7-5 Select sample from target audience / 

7;6 Validate mastery items . 

7.7 Select principles from learning theory and th^^^ psychology of 
. iearning appropriate to materials being developed ' 

7.8 Inform associated agenciesiof specilic developm^^^^ ^ 
via the Center Nev/sletter * " : 

8.0 Identify Key Concepts, Scope and Sequence for Entire Instructional Package 
9 . 0 Break Key Concepts Into Modules : " 



10 



0^ Identify Scope and Sequence of Each Module and Match Each Perfor- 
mance Objectiv e to Appropfiate Module • ■ : 
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?V5"'^'^''°V^P^'^^^y ^Pf^'"°P"^'^^ Instructional. Strategy 

#.¥^S,V.. .^I'l ^^'.^^'■'"ine a ppropriatr class, size ' ' 1. fi^'utlS' 

:' ■ ."• •■ . 11.1.1 Larg e . g roup . { 1 0 +) . . 

i?>M.V---." . 11.1.2 Small: group'-(2-9) • . 

r:;. U.1.3 Individuals ' ' -~ 

"" ll-.'2 Determine instructional format ' ' ' .. - - 



, 11.2.1 


Instructor 


required ' " • 




11.2. 1.1 


, lecture . - 




11.2. 1:2 


team-taught 




11.2.1.3 


symposium ' > 




11.2.1.4 


, audio-tutorial" 




11.2.1.5 


other / 


, 1112.2 


Instructor 


not required _ . 




11.2.2.1 


IPI ■ . ■ ' . 




11.2.2.2. 


CAI (or variants thereof 




11.2.2.3 


otfier . " 


Determine instructional media' ' ^ ' . 


11.3.1 


print 






11.3. i.r 


simulation ' , . 




11.3.1.2 


games . - . • 




-11.3.1.3 


case studies ; . 




11.3.1.4 


other . . . ' 


11.3.2 ^ 


non-print 




11.3.2.1 


CAI (or variants thareof) ' ' 




11.3.2.2 


audio-tape 




U.3.2.3 


video-tape 




11.3.2.4 


sllde-tapa . 




11.3.2.5 


. transparencies ' 




11.3.2.6 


video-^cassette 




11.3.2.7 


other . 



11.4 Bibliography 

12.0 Identify Resources Needed {development team, consultants, specialist?, 
materials, equipment) 
12.1 Make cost estimate - . ^ - ■ - ■ 



13.0 Estimate Time Required and Schedule Activities 
14,0 Write Summary of Context and Input Phases ' ^^^^ 
" 15.0 Submit Summary to Bevelopment Unit for Approval . 



43 



Choose Development Team(s); Procure Needed Materials and Equipm it 
16.1 Division of labor ' ^ 



16. 


1.1 


content 


16. 


1.2 - 


V evaluation .,, . ' 


16. 


1.3 


. media 


16. 


1.4 


instructional ■design 


16. 


1.5 


editor 


16. 


1.6 


progranimed instruction 


16. 


1.7 


CAI man 


16. 


1.8 


other 



Generate Organizational Plan for Development TWm(s) (develop PERT 
chart v/ith tlme-^llnes) 
17 . 1 Scheduling of development team members ^ ' 

17.2 Develop communication network between and within teams; 
between/among administrators^ developers / evaluators 

17*3 Develop mechanism for resolving team confUct/deficlenGies 
17*4 Inform associated agencies of development timelines in the : 
Center Newsletter - 

Construct Prototype 
18.1 Review all previous decisions made 

18,2. Review problem statement and objectives _ . 

18.3 Secure developmentar copyright from 0,Ev 

Test and analyze prototype 
19,1 Tutorial test ^ 
19.1. 1 Identify student from target population 
. 19 1 1,2 Administer components of the prototype 
19,1-3 Revise ^~ 
.19.2 Developmental TesV" 

19.2.1 ^ Develop evaluation criteria for test 

19.2.2 Identify group of 8-10 in common locale from 
target audienca 

19.2.3 Develop formative evaluatlon niethodology (and 
; Instruments, if appropriate) 

1^*2*4 Administer components of the prototype (develop- 

f mental testi - > 

19.2.5 Revise package (someone other than the author) 

19.2.6 Make adequate number of copies of the package 
19,3 Field Test 

19,3,1 Contact disseminating' agency re their Infor- 
mation needs 



, 19.3,2 Develop evaluation criteria 

_ 19.3.3 Identify five field test sites each with '8-10 

^t^dents and negotiate necessary arrangements . 
19.3.4- Develop summatlvQ evaluation methodology 

-and instruments ^ . 

19.3-5 Identify and train Instructors , if appropriate 
19,3,5.1 on-site ' 
19-3,5.2 at Center 
19-3.6 Administer package to studenti^ 

19.3.7 Collect summatlve data 

19.3.8 Analyse summatlve data ^ ' ^ 
19-3.9 Prepare field test report fdr dissemination 

with the pacj^ge ■ * 

• • # ■ ■ • •• . ■- ■■ . 

_ . . ■'•f ■ . - . . - 

Implement and Diffuse / " . » 

20.1 Send five completed packages to OE ■ 

20.2 Send five completed packages to PRDB ^ - ^ 

20.3 Send description of package to associated agencies, TAP.? atid-rotheL 
Select diffusion strategy ./-^ . 

20.4.1 Press coverage ^j"^' '■: 

20.4.2 Informational letters to CEDaR,%P, and a&D centers 
, 20.4,3 Samples to leaders In the field 

20.4.4 Other 

20.5 Assist PRDB with diffusion until termination of project ' 



Part Three: Testing the CID Modgl . . '. - - " 

Testing the usefulness of the model will involve a) securing profes- 
- Sional judgments as to its internal consistency and face validity; b) sec- " 
uring.the approval of the Center staff to Implement the model; c) systema- 
tically documenting the appropriateness and usefulness of the model after 
installation. 

Experts associated with the Center will be requested to evaluate 
the CID model in terms of its logic, completeness and communicability. 
Th^s^was done during the July, 1972 meeting of adjunct professors from 
associlatod evaluation agencies . ' ,» ; . 

Following revision of the model, based on the comments of the experts, 
the Center staff agreed to an experimental implementation of thi'model 
with review of its effects after one year. Implementing this model in any 
agency requires a) agency consensus on the appropriateness of the model 
as a guide for agency instructional development activity; b) commitment on 
the part of the agency developers. to the operationallzatlon or adoption or 
adaptation and utilization of a design for testing the model; c) comparison 
of the quality of products produced vls-a-vls ,the%6del with that of previously 
produced products; d) commitment of sufficient resources to instructional 
development. . 

. wMter Ipstallation of the CID model, the documentation of its useful- 



nBstmtn 



begin using three instruments," a check list, a log^ and a chart. 
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The^check list will contain items from the, model that are checked . off 
^ (and dated) as they are used or pursued. Figure 5 depicts the format of 
the CID check list. • - - ' " .. 

^^TE CHECK AS COMPLETED 
■ . 1.0 



1.1 

1.1.1 

1.1.2 



Figures: CID CHECK LIST " 

The Instructional development unit will use the Time and- Effort Log 
as shown m Figure 6. However, in order to Indicate more specifically 
what activities are pursued, the following code will be used: 



A 

B - 

C -i 
D -i 
E - 
F - 



"Evaluating the Educational Change Process" package 
- "Surv'ey^of Evaluation" package - 
Searching' for other packages ■ 
Project coordination (e.g. staff meetings) 
Center coordination (e.g. Center administration meetings) 
Faculty/College coordination (e.g. committee'meetlngs) 
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Date: Thi week beginning Kondiyp 



thru Sunday, 




'^"^^ Commenti ; 



5^ 50 
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; - ■ ■.. ■■ 

• .-V - . .,- ^ - - - • - ■ . . • 

The approprlatrietter be written In the cell alons with the amount of 
tln,e spent. Ihus, if the developer has spent two and 6ne-hM£ hou>s 
"searching for other packages, the code will be ■■2.5C". The search was 
probably conducted by "reading documents." "person to person consultation. 

"correspondence. " or "phone"--all of Which are headings abc55S a colunin 
.otcells. The appropriate cell is cho_sen and "2.5C" is written m. 

• ^" l"="^"cti°nal package is the subject of-the'actlvlty, 

I 'he Identification nu,„bers from the CID model can be. ased i^^^ 

atlon with the above instructions . For example . if the devgloper spends 
four hours in preparing the field test report for a certain instructlona^^^^ 
the code wflttenln the log is: "4A 10.1.2". ne appropriate cell, in ihis/ 
case, would be In the column under "Writing Reports." ; ; 

■ A wall Chart listing products (and dates) will be malntainedi'for the 
entire project. The prcducts could be added to the list dailV and include 
"mlnl-prpducts, " such as memos. Individual transparencies, and mockups . 
A-fomiat for the chart is' depicted In Figure 7. 

Product Date Product Date . -Product Data . 1 



2 

3 t 



s : 9 

S— _ 10 

7 ^ ll" 



8 ' 12 



Figure 7 1 CID PRODUCTS C H^RT 



;|r/;H-^lysls In order to discover common and variant criteria .which'can be 
•;%i'?P^^^^ -'.ppropriately to different types of instructions products for example/ 
^::;:lt may be possible to identify sets of criteria that will be. appropriate to . V:?; 



ai. development of two-day workshops as opposed to 30-hour 
' ■"• iimm packages. 



instructional 



- ; --A..y-ar after it is installed, the CID model's usefulness should be a 
matter ot record. If its positive utility is empirically established, the'' 
documentation effort would be refocuTed to address questions such as: 

1) Would other representations of the general model in the form of ' 
a PERT chart, a critical path network, a time pie, or a moving wheel render 
It more communicable? ...-.r- y^^'-y 

- 2) -Can the components of the model be displayed or coded in more 
appealing ways, such as color coding, problem cards, or gameboards? 
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.■ ORffiNTATION TO Ef ; 

■ . fiirpo'ses and Objec t ives of EPEC . . ' : j^-/;- 

' ■ - The initials "EPEC" stand for Evaluating the Process of Educational f::,'. -'^'^^3 
; ' : Change. There are three purposes to the EPEC system. One is to help you"..,;/ r:_.^^,^,l:, 
^ Identify the^klnds of decisions an evaluator services. The second iVto^pro^^^"^^^^ 
vide you with an experience" in evaluating the installation of a particular educa- 
tional change or innQvatlon, The third is to try out some specific process eval- 
uation skills . ^ ^ ^ .-^p^ 

The evaluation profession is one of the newest specialties in tthe^^-aald>;::Cl*^'-^^^^ 

of education. Its growth Is attributable to a variety of circumstances, not 

the least of which are the needs of the educational consumer. Taxpayers, i " '"""-i^T". 

j 

parents and students have, in recent years, become more insistent th"at educa- 
tors be accountable for their decisions. These consumers want to know how 
. cun'lculum^ trans portationv supervision and other Innovations are going to / 
benefit themv Thus / new university- and field-based training programs have 
V ^:b3en created to prepare skilled professionals who can provide educational 

consumers with the information they require. 
. r . Ftofessional evaluators must be able to provide many different kinds of 
information to their clients. EPEC is concerned with some of the skills and 
7 knowledges evaluators need in order to provide a particular kind of Information — 
V : Infonnation concerning the degree to which soma innovation is being success-* ^gs 
fully Installed, These skills and knowledges are communicated by involving 
you first in a simulated evaluation t^jsk. You will experience that task first V 5 

_ hand in a controlled setting. During the task you will play out the roles of =1^^ 



both the advocates and consumers of a particular educational change attempt. 
By playing these roles in a controlled setting, you will become sensitive to 
the significant influence Qdvocates and consumers have on educatlonQl innova- 
tion/ As a consequence of this sensitivity^ you should be better able to evaluat 
the process of installing educational innovations. Then, by negotiated contract 
with your instructor, you will apply your learnings from the simulated task to 
evaluating the installation of an innovation In a real educational setting. But 
now we are getting ahead of ourselves . 

The EPEC Instructional system provides opportunitles_|or participants to 
develop the foUov/ing competenclesi 

—describing variables associated with the advocate of an educationa 
innovation, \ 

"describing variables associated with the consumer of an educatlonar 
innovation* 

— desoribijig variables associated with the organlEatlon installing an 
^^■' 6 ^ _ \ 

*^deacrlbing variables associated with the educational innovation selected 
for installation, 

—describing the kinds of transactions which occur between advocates and 
consumers during ^e installation of an educational Innovation in an 
organization* 

_ . " " " ■ ■ ■ ■ • ... _ . .... J . — . ... . . ^ . .. . 

. ---applying the force field diagnostic technique* 

; ' : =ldentlfyin the kinds of process decisions an evaluator services . - ^ • 
-r^-- : — ^ and using process evaluation instruments. 



-analyaing and reporting major results in data collected; 
-utlUaing concepts and skills of giving and receiving feedback, 
-identifying and evaluating some group process skills as they relate to^ 
installing an Innovation. ■ 



V -writing a contingency evaluate the real installation of an' 

/ educational innovation. 

Nature of EPEC 

The EPEC Instructional system provides participants opportunities to ^ 
acquire knowledge, skills and techniques for evaluating the installation of an 
educational Innovation. It looks at only one ridge of that living mountain cLied 
"educational evaluation. The design calls for nine modules / approximately 
three hours each in length. Emphasis of the entire design is on participant^ 
practicing process evaluation skills . Continuous active participation is de- 
anded by using a simulated situation In which the participant assists a fic- 
titious project director to Install Computer Asstfi tad Tn.fn.^H^. rr-ATV fn a flc- 
tltlous school using the skills in EPEC. to groups of three and six; pa rtlclpa 
use group process techniques such as role playing and brainstorming to apply; 
and reinforce their learning. Prepared readings assigned at the end of several ^ 
modules provide subject matter for subsequent modules. Thr carrying out of ' 
the individual contingency contracts Is the only other out-of-class work re-, • 
■quired;- , . ..„' ■:,:,■ :. ; ^: . 

EPEC is concerned with £art of the problem solving process W^^T ' 
problem solving Is explained as a five-step process; " 



m 



: - ^ 1. Identify and define the problem B-4 

2, Identify alternative solutions to the problem 

3. Select the one solution that best fits the problem 

4, DistaU the selected solution . 

5. Evaluate the process of installation and the effec- 

, " tiveness of the solution • " 

EPEC deals with steps four and five, Installing and evaluating a selected 
problem solution, it is linportant that you understand the focus of this system 
at the outset, * ^ 

It Is also important for you to understand that the developers of EPEC 
believe 1) that group Interaction is the most appropriate way to communicate 
to you the systematic observation skills necessary for evaluating the process 
.of educational change; 2) that EPEC 's content will become part of you only if 
you practice your learning in both simulated and real. settings , Thus, the EPEC 
modules will Involve much group activity and practice. 

You may experience some discomfort, frustration even, as you adapt to ^ 
this style of learning. The time allotted for group activity may seem too much 
■ ^oter' as you proceed through the modules. This is to be 

expected. After all, you are now the consumers of an educational innovation. 
The insfructor will listen to your concerns , but the developers have asked him ■ 
not to make any changes in the sequencing and timing of EPEC module activities. 
■ The developers will listen to your concerns/ too, as ySui record them on the EPEC 
Evaluation Forrh you receive at the end of- each module.' I 

STOP 



* Participants who desire additional training in steps one, two and three of the " 
problem solving process should arrange to take Research Utilizing Problem Solving 
- (RUPS), an Instructional system developed by the Northwest Regional Laboratory 
( I /--Portland^ Oregon, B7 ■ ' 
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EPEC AGTIVITIES - MODULE i . 



DESIGN: 1. Purpose 
' 2. Rationale 

.3. Performance based objectives (level 1) ' A 

; 1. CALENDAR FOR EPEC MODULES 

Purpose: Arrange meeting time to cond 

: Rationale: Positive effect on modules 1£ the partlclpant^^c^^ 

detefmlne the time most suited to his scheduie 
for the modules 

Objective: EPEC participants will agree on and Individually 
record on thD proper page a schedule for the'nlne 
EPEC modules. 

•2. ORIENTATION TO EPEC / 

Purpose: Give the participant a perspective of the package 

Rationale: Establish a common frame of referenee to facilitate 
acquiring the objectives of the course 

Objective: Participant will correctly identify during the Self-test 

1. the purpose and objectives of EPEC • 

2. that EPEC s method of instruction focuses on group 
process techniques 

3. that EPEC content emphasize the development of 
process evaluation skills 

3. TAPE PRESENTATION AND ELEMENTS OF INNOVATION INVENTORY 

Purposei Present the participant with an organizational frame- 
- r "^^oi^ for processing Information o 

innovation 

Rationale: Necessary competency of an evaluator 

Objectives: 1 . The EPEC participant will identify the following 
variables presented in the taped presentation 
by recording them on pp. I-ll, I-12 



a. the problem - 
. b. the proposed solution 

c, the arguments for and against the p 

■ ^ solution ; : - r ■ 

d, . the advocate, the consumer/ the Innovation, 

- arid the organization in which the innovation 

wUl occur 

e, the role of the aval ua tor in tenns of these. 
^ Interacting variables 

2, Participant experiences the dynamics of working 
In a team to further clarify the problem condition 

GUroELINES FOR WmTING A PROBLm STATEM^ 

Purposei Give the participant guidelines for writing a problem 
statement 

Rationale: Give the participant functional rules which he can 

Immediately use to dilinate the problem conditions and 
than further develop to creEte the system best suited 
to his role as an evalutor ' 



Objective: Participant will write a problam statement (1-^18) ' 
according to the ■*Guldellnes'V given on p* I-^l^ 

DISCUSSION OF PROBLEM STATEMENTS 

Purpose: 1, Discuss their problem statements 

2* Build teamwork skills ' ■ 

RaUonaler 1, Discussion gives the participant feedback on his 
Interpretation of the material 
2. Teamwork skills are: — 

a, necessary for meeting the objectives of EPEC 

b, necessary competancy of a process avaluator 

Objective: Each participant will modify his written problem statement 
as a result of Input from the trio discussion outlined 
on p. 1-19 

FORCE FIELD ANALYSIS 

Purposai To give the participant a tool for analyzing the forces that 
■ \ ^ have created a given' state 

Rationale: To obtain a better understanding of t^^^^^^^^ to 
facilitate reaching the goal state :^ ? - . 



ObJecUves: The participant will identify b correctly responding 
! . - to the self-test (1-22): 

a. the purpose of FFA; 

b. the manner in which the "perceived situation" ' 
. . , is represented In the FFA. 

Participant will perform force field analysis on his 

problem statement using the "Guidelines" given on 
1-20, 1-21, and 1-24 . 

7. DISCUSSION OF FORCE FIELD ANALYSIS 

Purpose: 1. Discuss force field analysis 
2, Build teamwork skills 

Rationale: Feedback facilitates learning 

Objective: Particlpantwlll modify his written problem statement 
as a result of input from the trio discussion outlined 
on p. I-2S 

8. ORIENTATIO^^ TO CONTMCT AND RULES FOR BRAINSTORMING 
Contract 

Purpose: State clearly for the participant- what he is ext.. 
to do for his final project 

Rationale: If the participant knows specifically what he is ex- 
pected to do he will be better able to work independ- 
ehtly on the project, as opposed to being dependent - 
on the instructor for direction and control of the project 

Bra insto rming 



-Purposej Give the participant a tool for generating a lot of ideas 
In a short amount of time 

Rationale: An aid for diminishing the probablllity of getting bogged 
down in the. initial phase of a problem 

1. Participants follow "Rules for Brainstorming" 

2. Each participant identifies one contract topic^^^^^d^^ 

the brainstorming session for the final project in 
the course 

3. Participants wUl identify the rules for brains tormlnc 
by scoring at least 50% on the self-test (1-30) . 



OSJECTIVEi FOR READINGS LISTED IN PACKAGE^: 



EPEC AGTIVITIIS - MODULE 2 



DESIGNr 1. Purpose - 

2, Rationale 

3, Performance based objactlves (level 1) 



1* IN-BASKET KERCISE # 1 



sei 1* To place the participant in a simulated ^^d 
making role 

2 . Call upon the participant to make decisions based 
on the situations structured In^EPEC 



Ratlonalei The; simulation will sens^^^^ the 
role of dccislQn maker bringing insight to his ro 
as process ©valuator 

Objective: Participant will record a decisionJor each in-*basket 
Item on the Action Matrix 

ACTION ANALYSIS PROFILE 



Purpose t 
Rationale: 



Organised form for listing possible actions token at 
a decision^ point 

To add to the participant's unaerstandlng of the de- 
cision role by alerting him to alternative deiSslons he 
may not have considered ^ 



Objective: Participant fills out the "Action Analysis Profile" 
1^4EANS OF COMMUNI^^ 

Purpose: Organized form for listing possible channels for 
communicating Information following the decision 

Rationales To add to the participant's understanding of the de- 
— clsion role by alerting him to alternative channels of 
communication he may not have considered 



ObJectl\^e: Participant fills out the "Means of Communication 
Profile'' 



4 . -DISCUSSION OF DECISION ALTERNATIVES 



Purpose: To have the participant experience the different solu- 

Uon strategies that can be generated a group of 
people to the same problem situation - ' ■ 

Rationale: Awareness of alternative solutions will increase the 
^ number of potential decisions in the thought process 

of each participant 

Objective: Each participant will present his alternative solution 
strategies and will respond to those presented by 
his colleagues, according^ to "Guidelines" on p. 11-15 

j. EVALUATION DECISION MAKING AND INSTITUTIONALIZATION 

The participant will read EDM and I (11-16 through 11-20) . 

. PROBLEM STATEMENT AND FORCE FIELD ANALYSIS REVISITED 



Purpose: 



Rationale: 



Objective: 



Reinforce the framework previously given for analyz- 
ing a problem situation 



Practice in using the framework will increi- 
skill level for use in the field 



2. 



Participant will revise his original problem 
statement using p. 11-24 to record his revision. 
Participant will revise his originol force field 
analysis, using p. 11-24 to record his revision. 



FISHBOWL ACTIVIIY 



Purpose: 



Rationale: 



1. 
2. 

3. 
4. 

1. 
2. 
3. 
4, 



Organize group process for obtaining feedback 
Identify some of the elements and relationships in 

the group discussion through the use of assigned rol 
Work within a time limit 

BuHd teamwork skills 

Feedback functions to develop sk.Uls and increase 
productivity of both the Individual and the group 
Assigned roles aid a participant to focus on a 
specific task for the group 

Working elthin a time limit assists the participant to 
focus on prime elements first " ' 

Teamwork skills are meant to. increase productivity 
and facilitate the solution of problems 



■ - . C-6 

Objectives: 1. Each participant will implement the -"Guidelines" 

{11-25/ n-26) for the Flshbowl Activity. 

2 , Each participant will express satisfaction with 

the helpfulness of input received during the flshbowl - 



I 

t 
i 

i 
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EPEC ACTmTIE - MODULH 3 



C"7 



DESIGN- 1, Purposa 
2 * Rationale 

3, Perforniance based objectives 



1. IN BASKET DCERCISE #2 



Purpose: 



Rationale: 



1, Use the participant's understanding of the situation as 
a resource to create en Qn-^the--spot prciblem incident 

2 , Process infomiatlon from observing a problem incident 
Into a problem statement 

3, Reach consensus on adequacy of problem statements 

Use of simulation as a training technique to practice process 
evaluation skills 



Objectlvar ^ l. 



2. 



The participant will formulate a problem statement ^n 
p • III -^6 based on Inputs from the In-^basket exerclsi 

The participant will reach consensus within his trio on 
technical adequacy of his problem statement-^-accordirig 
to the Inetnictlons on p. III-7 
3. The participant will reach consensus within his tdo on 
the content adequacy of his problem statement--acc6raing 
to the Instructions on p. 111-7 

2. REVISION OF THE CONTRACT . .. ^ 

Purpose: 1. Obtain feedback on their rontracts 

2 . Use new knowledge to revise contract 

Rationale: Emphasize contract as their immediate vehicle to be used in 
applying knowledge as it is gained in EPEC to a practical 
situation . > • , 



Objectives J 



1. The participants will discuss their contract with other 
members of the trio according to the guidelines on p. III-8 

2. The participant will revise the contract (III-9, III-lO) - 
using Input received during the trio~3lscussion 



OBJECTIVES FOR READINGS LISTED IN PACKAGfe 



EPEC ACTIVITIES - MODULE 4 



DESIGN I 1 . Purpose 

2. Rationale 

3. Performance based obJectlveE 



1 . CHECKLIST OP INNOVATION CHARACTERS TICS 



; PurppFe: Apply Brlckelr and Havelock's fremework for analyzing 

characteristics of an Innovation to the CAI Innovation 

Rationale: To provide a useful technique for analyzing Inno 
which the participant will encounter in the field. 

Objectives' 1 , The participant will fill out the "Checklist of Innova- 
tion Characteristics '^(p,IV-6) as per Instruct^^ 
2, The participant will vocalize no objection to the scientl 
\ fla status of the EPEC Instrument* 

Z; ^DISCUSSION OF INNOVATION CHARACTERISTICS 

Purpose: Compare and discuss responses made on the checklist/ 



Rationale: Participant needs feedback to clarify and strengthen his 
Interpretation of the material ' 

Objectives: 1* Each participant v^ill give and receive feedback concern^ 
- ' ing responses to the checklist 

2, The participants will reach consensus concerning those 
innovation characteristics whicy apply to CAI 

3, IN-BASKET EXERCISE #3 

Pun^ose: 1 • To place the participant in a simulated decision making 
role v. - • ^ 

2* Call upon the participant to go through the decision 
making process 

: y 3, Analyze the decision making processJn terms of the pro- 

-V files given in EPEC ^ ^ 

: ^ Ratlona to the decision making process 

i^^ ObiQctlyel 1. The participant will make a decision on . each in-baskef v; 

item and^^record the decision on the"^"Action Matrix, 
p. IV-16 - 



2 . The participant will transpose his strategies to "Action 
Analysis Profile" and "Means of Communication Profile" 
pp. IV-i7— lV-19 

3. The participant will complete the "Situation Analysis 
Profile," p. lV-20. 



4 . DISCUSSION OF DECISION ALTERNATIVES 
Purpose: 



1 . To analyze different solution strategies In terms of a 
common framework (3 analysis profiles) " 

2. To work within a time limit 



Rationale: 



1 . Awareness of alternative solutions will inrease the 
numbef of potential decisions in the thought process ' 
of each participant 

2,, Common framework will facilitate assessing the adequacy 

of each strategy 
3 . Time limit for presentation should force participant to 

focus on key elements first j 

4. Working within a time limit should aid In keeping parti- 
cipants on task • ' 



Objectives; 



The parUcipants will present and analyze in trios alternative 
strategies using the analysis profiles on p. IV-16— IV-20 



. PROBLEM STATEMENT AND FORCE iFIELD ANALYSIS IN-BASKET # 3 

Purposei 1. Sttangthen these skills in the participants repertoire of 
process evaluation skills 

2 . Stress these skills as the first steps to perform in 
analyzing new information 



Rationale: 
Objectives; 



Necessary for process evaluatlorir ' 

1. The participant will write a technically adequate problem 
statement (p. lV-22) on information obtained from in- 
basket #3 Items • ^ 

2. The participant wUl perform a force field analysis of ' " 
problem statement on p. IV-2 2 



. F 1 SHBOWL AC Tl VIIY 



Purposei 1 . Organize group process for feedback 

2. Help participants analyse team processes 
■ - ■ ■ . 3 . Build teamwork skills 



Rationale: Necessary competency of process evaluator ^ ; 



Objectives: The participants will follow the Instructions for flshbowi 

activity (p. iv-23) using their problem statements and 
force field analyses as topics - 



REVISION OF CON TRAaT - . ^ 

Purpose: 1. Revise contract based on new knwoledge gained frim - 

2 . Provide the participant with feedbaGk on his contract 
from members in his group 

Rationale^ , Further use of the contract as the vehichle: for tying know- 
ledge as it is gained to an actual innovation 

Objectives: 1. The participant will revise the "Innovation" section of his 
contract, 

2. The particlpant.wlU discuss the •'InnovaUon" sectioh^o^^ 
his contract with the members of his trio ' 



OBJECTIVES FOR READINGS LISTED IN PACKAGE 
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-EPEG~ACTIVITIES MODULE 5 



DESIGN: 1. Purpose 
- ' . 2 , Rationale 

3. Performance based objectives 



1. SUMI'-IARY AND PERSPECTIVE I 



Rationale: 



Purpose: 1 . Give the participant a check point to match the intended 
objectives of the package with what he has learned. . 
If there is an apparent discrepancy he should be able to 
remedy it by going back over parts of the package. (The 
-EPEC Activities" can serve as an aid for the instructor 
in helping the participant locate the relevant activities ) 
2 . Bring the participants into the same frame of reference * 

for proceeding wlthjhe rest of the package 
3. Give the participants an overview of the model on which 
EPEC was designed _ | 

1 . The check point reinforces- those participants who hlvll 
put the necessary work into the package. It allows'^ "^^^^ ^^ 
those that haven't an opportunity to catch up and attain 
the objectives of the first three modules, which are pre- 
requisites for the rest of EFEC 
■2. Common reference point or perspective is'meant to help ? 
the participants attain the objectives for the following 
modules " 

3 . Giving the participants the module on which EPEC is based 
provides them with - one way to assess the value of the 
package and the rationale behind it. 

l._ The participant will read "Sumrnary and Perspective! " 
7. . -The participant will correctly Identify in-the: Mastery : 

Test a)„thatjhe "second half of EPEC is concerned with 
. the advocate and consumer rather-than the organization 
and the Innovation; b) that observing and reporting ad- - 
~voeate-consumer-behavlors-are-ff neces saTy skill of "the ^ 
process evaluator—that memorizing and reporting the be- 
havioral categories is not sufficient. 
3. The participant will write out the three purposes of EPEC 
on the Mastery Test. 

INTRODUCTION tFroLE PLAY AND ROLE PLAY ICEBREAKER 



Objectives; 



Purpose: 



1 
2 



Explain purpose and technlqxre-of-rol'fe playing- 

Glve all the participants a chance to practice the technique 

in a simple exercise - 

_ B9 , 



'^Rationale: 




Objectives: 



.3. 



.the techn qu^ because they will be using it as aWi^rtfel 
toll in following activities , ' 

1. The participant will read Introudctlon'io Role n^^f4^>i 
The participant will participate in the role play ic;br;aMi« 

.ROLE PLAY.CALENDAR ' - - - " ■ ' " 

; .Purpose: Solicit „^iu„teers for .he role playing ac.iviUes ' " ' .^^^ 

Rauonale: Havin.^.he rples^ssi^ne^ in advance sK^M help '.e ecUvftllf 



f -1 



4 . ROLE PIAY 
Purpose: 



1 



2 
3. 



Rationale: 



:ObJ actives: 



To provide consumer and advocate the .opportunity oJ 
simulating an actual confrontation ? 

?o ^mi;'H°".H^' consumer resistance to an innovation 
To provide the remaining participants an expenonre ln 
observing a consumer advocate confrontation 

lT"-Simulation is an effective technique for actively involvlno 
participants In the learning experience ^^^olvlng 

unLrsTanSff" f Participants in their 
understanding of systems they will encounter in the 

The participant portraying the consuS^TwlH eni^^u "7"~ 
res^i^stan_ce characteristics based on tt.e Situation st^d"^^ ' 

The participant portraying the advocate will trv to spII th^ 

resistant behaviors he is exhibiting 

The participant serving as" process evaluator will record obs^rv. 
tions of consumer behavior on practice sheet supplied 
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5. GUIDELINES FOR DISCUSSION OF FIRST ROLE 



Puipose: 



Structure the discussion so that: ' 

1. The participant playing the consumer gets feedback on th. 
consumer behaviors he Illustrated . ie_eanack on the_ 



Rationale: 



Objectives I 



picked up - - - ---^';^^^^,^ lns^o^r^^s■ 

^ ' fh'l't™^"^- ^^^^'^ consensus among the' observens^^^ 
the behaviors recorded * . 

4. Analyze any mismatches in the points above " -r V^^ 

^' oh^f °" l^ n' ^^'^'^^'^^^^ 3^^l3t them in building" v 
observation skills . \ ^ -/ 

^' nl^^ir^'i'^"^''' structure assists the participants t^^fbbW- 
tL)rl '^.""''^^^ °' exercise, while guarding agalnst" 
their spending too much time on insignificant points ;• V;.../' 

(prV-T/.f '""^ Guidelines for Discussion 

^' Spr f f'P^"' ""^ '""^'^^'^ the mastery_ test that 
leamin t ^"^ checklists are Interide.d only as 

learning tools--not as validated instmments 



ATTITUDE INSTRUMENT AND TRIO'ACTIVITY 



Purposer 



Rationale* 



Objectives: 



f^ltrT r-l^f P^"' ^" opportunity to assess his MitJe - 
Ss wo " ' -"''■discuss it wlth-tha-other mo.nfclrs o£ 

- :1 the partialpants more clearly understand their' feelings 
towards pla playing , it should assist triam in anafyz no 
future role playing activities - \ ^V-mq 

^' ?pf V-Ts.f ^^""^ ^"^^ complete the "Summary Instrument" 

2. The participant will discuss with his tllo his responses to J 
the summary Instrument . V , r ^''^^^ — „; 

3 . The . participant will, verbally reacVto the surnmary instrument 
responses of the other members oLhls tr\o. instrument^ 



ROLE PLAY # 2 



Provide the participant an opportunity to study the role of 



Rationale: 



Objectives: 



Effective training technique for building the skills n 
process evaluator 



ecessary to a". 



The participant role playing the evaluator will use a stntPou 
So nam f T^'^ ^" accordance with role "ay caTd (p^v-1.) 

tne evaluator . The participant role playing the observer will ■■ " 
complete .an-ovservatlon sheet (p. V-21 ) - ^' 



■s® ... . 



•2:- '^^\^;y'~^i^^^Tlh 



^^•^Mg^ideiAnes for discussion of role play # 2 

Structure the disciission 'SO -that: 

1. Participant piaying. George High ;getsn[iSfc/S!f^ 

strategy he used, to service. a Ueclsion maker •'".^■V --^^'T-J-lsA' 
.2-. Analyze the strategy used by:.the role player and possibl^M 



T;V'.:' 



^ a.ternative strateglGs^in terms of the concepts and^skill^^lM 
taught in EPEC ^^M^^^^y^,^:: 



Rationale; 



Assist the participant in his conceptualizing about the^li?®^ 
of a process evaluator \ ^ riv- ^ - --m^^- 



Objectives: The partlclpants.will follow guidelines for discussion (p; V-22)...„ 
9. CONTRACT- REVISION ' ^ 



Purpose: 



Rationale: 



1 . Thxough discussion obtain feedback on present state of . . ' 
contract in terms of interview guidelines ' "■^"'^ 

2. Begin revision of contract 



1. Feedback from other members will be useful in 



contract 



revis 



n 



2 . Begin revision while feedback from 



clpant's mind 
Objectivesi 1, Participants discuss contracts 



m group is fresh in' parti - 



Participants begin revision on their contract to b 
in Interview ' 



e presented, iv; 
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EPEC ACTIVITIES - MODULE 6 



C-lSw 



DESIGN: ^ H Purpose . . 

-r. 2 . Rationale - 

3 . Performance based objectives 



1. ROLE PLAY REVIEW- 



Pufposei 



Rationale: 



Objectives: 



2. ROLE.PIAY# 3 
Purpoae: 



Rationalar 



Objectivesr 



Sensitize the participant to the complexity and inter-relation- 
ships of variables he Is attempting to delinate In his obs 



tlons 



erva- 



Noting and discussing discrepancies in his own observations 



1. 
2. 



The' partlclpant-wiir list as the tape Is played evidences of. ' 
consumer resistance on the QRB Practice Sheet, p. ^1-5 - 
The participant will discuss in trios theL^-responsed to the 
Practice sheet and the discrepancy between those responses'! 

Module i , according to ■'Guide-^6 



and th^;g|sponses made .in* 
lines/;?;. p. Vl-lir^f^^' 



1. Use the participants' understanding of consumer and ad- 
vocate roles to^simulate a confrontation to study consumer 
.acceptance behaviors . „ uaumci 

2 . Build observation skills ' 

1. ^.Awareness of consumer acceptance behaviors provides 

useful _lnformation for implementing an Innovation 

2. Observation skills are useful to the process evaluator 

1 . The participant playing the consumer role will portray 
consumer acceptance behavior consistanc with the situa- 
tion structured in EPEC ■ • • 

2 .- The participant playihrWtavocate rolT wlTrt^^ toli^T"''" 
more sup^^fcr the innovation based on information given 

_ about the^cprfsSmer and,:the behaviors he Is exhibiting " 

3. The participant playing the process evaluator will -record 
observations of consumer behavior on instrument supplied 



3. . GUIDELINES FOR DISCUSSION ROLE PLAY # 3 



C-16 



- -f, 



Purpose: Structure the discussion so that: / ,'. ' . :-' • 

The participant playing the consumer gets feedback' on - 
the consumer behaviors he illustrated ^/ " 

2. Determine if there is a match between the 'behaviors he -" "-^^ 
intended to illustrate and the behaviors the observer picked" 

up 

3. Determine if there is consensus among the obsen/ersTOn T 
the behaviors recorded . • 

4. Analyze any mismatches in the points above 

Rationale: 1 . Feedback on their observations assist them in building M 

observation skills 

2. Guidelines help them focus on the objectives of the 
activity 



Objectives: 



The participants will follow tte '•Guidelines for Discussion'' 
on p. VI-12 



ROLE PLAY # 4 



Purpose* 



Rationale* 



Objectives: 



1. 
2. 



Demonstrate- that It is likely that there will be both forces'' 
for and forces against the innovation operating in a consumer 
Strengthen observation skills of participants ■ 



Necessary to get a profile of the consumer that is as complete 
as possible for Installing an educational Innovation 

1 . The participant playing the conaumer'role will portray both 
consumer acceptance and resistance behaviors toward the ' 
Innovation ' . 

2. _ Ti)e participant playing the advocate role will try to gain 

more support for the Innovation based on knowledge of CAl' 
and consumer behavior 

3. The participant playing the. process evaluator will record - 
observations of consumer behavior on instrument supplied 



GUmELINES FOR ■DISCUSSION-ROLE"PLAY'#~4' 



Purposej 



1 
2 



Feedback on the role play'actlvlty 

Recogriize that In focusing on both acceptant and resistant 
behavior, they have put Jogether.a force field analysis of a ^- 
consumer's position which will need to undergo change 
before the goal state is reached 

Discuss forces for and forces against during the role play 
in terms of their affect on instailirig the innovation 



Rationale: A complete profile of the consumer in terms of a force field 
analysis has direct'implicatlons for formulating specific 
strategies to implement the innovation 

Objective^:.. The participant's will follow theJlGuidelines for Disfiuision" 
on p. VI- 18 - -v;f^ 

SCHEDULE FOR CONTRACT AND INTERVIE.\^* GUIDELINES 

"Purpose: 1. Schedule individual contract interview 
; ^ . / :v 2 . : Clarify; the task -by presenting-guideU for the interview 

. 1 , „Fonnally obtaining agreGmont on an interview date puts • 

V the responsibility on the participant to be prepared 

2. Guidelines vvill assist "both the instructor and the parti^ , 
cipant to focus on- performance relevant to the task ^^^^^^^^- -^ 

- Objectives^ The ins^uctor -and the participants will agreevon interview 
dates a^nd ^v ill record those dates p. VI-20 

CONTRACT REVISION " - . ^ - 

Purpose: 1. Through discussion obtain feedback on present state of 
contract in terms of interview guidelines 
2. Begin revision cf con^'act 

Bationale: I v Feedback from other members will be useful In revising 
contract . " ' ' ' 

- - 2 . Begin revision while feedback from group is fresh in 
participant's mind 

Objectives: 1> The participants will discuss contracts in trios using 
Vlj-ig as a discussion guide 
- 2 * The participant will; begin 

P^es^nted in the inten/iew with the EPEC instructor 
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EPEC ACTIVITIES " MODULE 7 



:':.DESIGN: 1. Purpose 
: s . 2, Rationale 

"/ " 3* Performance based objectives 



l; ROLE PLAY # 5 
Purpose: 

Rationale: 

Objectives: 



Use simulation toTstudy the advocate strategies for brlngili|Sffi| 
about adoption of an-innovation as proposed^n'EPEC ^ v^^^^^^llii 

Training technique to assist the participants in assimulating'"^ 
the strategies presented in EPEC ' ' 

1 - The participant role players enact the role play characters-'-^l 
described on the role play card. " ' .^/-.-rff?^ 
2 . Th participant s erying ' as . prbces s"- ^'i^aTbators 'will record'^S 
their observations-of the-role play using the "Advocate ,.-->^ 
Strategiei;.. Practice Sheet" (p. VII- 7) 

GUIDELINES FOR DISCUSSION - ROLE PLAY # 5 
Purpose: Analyze role play focusing on advocate strategies 



Rationale: 



Objectives : 



3- ROLE PLAY # 6 



Purpose* 



^r^^Ratlonale* 



Objectives! 



Lead to a clear understanding of the advocate strategies 
proposed in EPEC 

The participant leader and discussants will follow the "Guide-^ 
lines for Discussion" on p. VII^S 



Use simulation to integrate the concepts for advocate 
strategies with consumer resistant behaviors 

..Method for^analy^ing^ interaction ^batween-consumer-andrad,——^ 
vocate during the change process ^ " 

1, The participant role players enact characters based on in- 
formation given them 

2. The participants serving as process evaluators record thelr""^"^^ 
' observations of the role play using three practice sheets 

(p. Vn-12, 13, .14) 



IGUIDELINES FOR DISCUSSION - ROLE PlAY # 5 



.C-19 



-Purpose; 



Rationale; 



Objectives; 



Analyze the interaction between advocate strategies and'cb^f 
TzPeT consTrucTs 

Necessary for the participants to understand the eolation- 
order for it to be functional for them in the field 

The participant leader and discussants will folJow the "Cuid^ 
lines for Discussion" on p. Vlt-lS 



GUIDELII^ S FOR DISCUSSION OF ADVOCATE STRATEGIES- 



Purpose: 

Ratlonalei 

Objectives! 



Participants receive feedback from each other onthe use of. 
advocate strategies in their contract ■ 

Enable the Participants^tb make thVlnost effective use of 
advocate strategies for installing their innovations 

S^eyS'Tn1h%"''' '1'"""^ ^^'^^^^^ias which 

on ^. VIl!l6 following discussions guidelines 



EPEC ACTIVITIES - MODULE 8 ' - ' - .'Gt^oI, 

~ ^ .;■vv^;■■[^T:.wV^:■■^ ., ■ . ■ . • , . , 

f ''^* ' ' ' e ' - ' * ' ^ , ' S: 

:-.':DESIGN: l. Purpose- ' ■ ' 

- ' - - 2. Rationale • ' -•.•'j' 

- - 3, Objectives " '--7 -I 

-SUMMARY AMD PERSPECTIVE II . "".-rl 

Purpose: l . Review tire Instructional model on whfdh EPEC.was ' ~ ;f 

2. Assist the 'participants to synthesize the skills and 
concepts for change presented in EPECf • - - 

3. Increase the probability that the model for 6hange - 

cZnT . ''f "^'^ ^"^^^ installation of the 
cnange designed in their contract 

Objectives: ^^f^r^ipant win reil'-Summifv^and 
2 . ELEMENTS OF INNOVATION INVENTORY 

Purpose: Reinforce the framework.glver, in EPEC for analyzlno 

an innovation ~ - - ^ -^y^^ng 

Rationale: Practice in using the framework win increase the sHll ' 
level for use in the field "le skui 

Objectives:. The participant will.complete the inventory (pp viII-5 
through VIII.7) based on the simulated CATin'ovaJion 
3. -IN-BASKEI EXERCISE # 4 AND DISCUSSION OF DECISION ALTERNATIVES 

^Purpose: ^^^^m^ ?mechanisnaor_detern.ining.the 

effect or EPEC experiences upon their decision raaking style 



Rationale; 



Aj.areness of changes in the participants' decision maHna 



Objectives.. 1 . The participants will complete the Action Matrix. Action ■ 
Analyses Protile, and Means of Communication P om" 

■ n^o.fd"H"'^"'^f to the instructions ' ' " 
provided on pp. VIII-1 6 through Vm- 19. 



2. The participants will discus's 'the ef&i-EpfcT,^^: h^^^^^ 



4. R^IS ION OF THE CONTRACT ' '..-v . "I r -. ..--'^-VliS 

' • " , •' '-r'-^^H 

, ' Purpose: 1 . Get final feedback on their contract from group ' ' " " ^"^t 

members before implementing their innovation .-./VV? 
^ • f ^°"tract- for possible problem areas ' ■ 

m implementation " . ' . 'V''-, 



„ Rationale: 



!„"Srau;f """^'^""^ f---ccesT-i„ .nstallii^:;-^ -^^ 



Objectives: 1. The pamclpant will aiscuss his contract and ^^/dbwil^ 

dX °/t^o prr^' ---laijivi^l 

^' JovlsT„'.'i°H''"' T" '"^ instructor's'consont for ahlS 

'""'^'""^ ''^^ '° in his contract at-thlsvame^ 



5- . ,S. 1*-' 
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EPEC ACTIVITIES - MODULE " 9 



C-22 



r^^DESIGN: I. Purpose ' 

2. Rationale . -If iCcf 



. -3. Objectives... ; - • _ iVferfv/.'-. 

1'.. ORAL REPORT AND ASSESSMENT _ . 

. ,;. Purpose: i . . Have the participant present to the group the.outcomeS 

. .„._ofiiis,_c.ontract j_.-_„..._„,^ _2_._-.-i: 

2 . Have participants perform thn, external evaluation of' 
individual projects; based on the InformaLaon given ' ' 
tnem in the presentation 

Mechanism through ^Ich the participant demonstrates I 
accountability for his contract ' ^ f 



Rationale: 



Objectives: l 
2 



Entry 
Behavior: 



of'hl^conttct ^""^^ -^-^ - the; outcome 

uslTthf o'f the oral reports of his pe^rs 
using the Oral Report Assessment Form , p. IX-3 , - 

P"tlcipant-will use at least five EPEC idios.ynciatlc " 
terms in his oral report'. ^ "-^'"-'^ 

Sfli^'b'^g^'ns'" "^f"- 



2. GUIDELINES FOR FINAL DISCUSSION 

Purpose: 1 . Enabirtha participant to discuss the evaluation ha 
received on' his oral report with the evaluators 
^. Provide the participants with the opportunity to further 
explore any of the o^t comes of the contracts 

- , _ - - ^ • bhare perceptlonsof EPEC - : _ , j 

_:.%Uonale:i:_l .Clariflcbtlan of-the- evaluatlon^repcrt^lnc-reaii?lfir ' ^ ■ 

relevance to the participant • 
2 . Verbalizing their perceptions of EPEC should assist them 
In evaluating their. "EPEC" experience 

Objectives-- The participant will follow the L-GJHdelln^s foTilnal 
Discussion" on p. K-S '^^"^^^ 




.-• ■ I - 




APPK^DK Di INSTRUMENTS 
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BACKGROUND INFOflMATION 



Sex?- 



S^' 3'5K^.^"'^';^^ Occupation 



.Age 



_ Yea rs PrcrfessiQnalSE>^ 



Institution and Location. 
Telephone: Business^; 



What is your educational background (Check all that applrf 

-rrzrrzrHigh^School'g fad uate — ' ~ 



^Some CQllage 



........... 



.College Graduate (maJor))_ 
.Masters Degree {major)' 



^Ph* D, Degree (mojprK 



_„ (degree) _ 
. Jdegree) _ 

(degree) 

Jdegree)__ 



' ^ Current Student (maJa-T_ 

(area of speclalizQtloj^___ 

How many quarter hoiir^ if th., u ~ 

rter hours. If any, have you completed beyond your last 

degree? 

-H-a previously participated In a workshop or course mvolvlng any of 

the following? (check all that apply)_ ' 
^ Role Playing . . . _ • 

iBralnstorrningT ""1"^^' r rr— j: ; - - .. ..... 

.Sensitivity Training ■ 







1?J1^ 1: Q ha ng e Th e cry 



.. - Why are you taking this course ond^what do you expect to gain 7 
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DIRECTIONS FOR ADMINISTERING THE PRQBf^JUESIldNNAIRE 

_ : During a. training experience participants ma7;feci^iently to each 
particular LA. Hence, If reactions are assessed at'th^eid of a module of 
Ws „,„ch info,™ation tends to Be lost due to forgetting or confounding. 

However, if participants must complete instruments after every activity too 
much time may be devoted to evaluation, participants may react negaUvely 
to the evaluation or the evaluation may interact with training producing 
different outcomes than would be obtained by training alone. The PROBE 
guestlonnalre is designed to tap participants, reactlons to each activity on, 
the spot With a minimum of Interference and time. It is Indeed a gulcfc f 
"probe" Of participant reactions to training in terms of a very few but critical 



dimenaions , 



The PROBE questionnaires, one for each module, are attached to the 
mslde back cover o, the participant's workbook by means of a spring clip ' 
The PROBE General Instructions is included in th. workbook as the last paper 
to serve as a ready reference as participants need to check the meaning o, 
the PROBE guestionnalre. ~ - , 

Before trainl.ng begins, participants are instructed to read the PROBE 
General Instructions, and ask gues.ions of the instructor.. Then, a. times desig- 
nated by the instructor, paniclpants are .-.sked to complete .ROBE for a specified 
activity . The instructor's agenda contains the cue. "PltOBE, " 1„ stamped - 
red letters, whenever a probe Is to be made.durtng training. Upon completing ^ 
a step preceding the cue. the instructor gives ;artic..pants the following ;/ . 



directions i . ... 

1 1 z= 83 



Pmm^a^^'l-!° ^ - °' your wofkboofc and coinplete ' 
HUH E for learning activity 1* r*Th« ,« * ^"'npiete : , 



" r ' ■■ : ■-■ ■ 'PROBE: 1) ■ ' ■ 

So.a dal.y n,ay b. expected the firs. o„o or two .i..sp.hic,p'an.'s 
co.p,ete P.0B.,b„^.3ne„„,u . 3 expected tH. ..,^^^^^^^ 
"ithln a minute of the directions quoted above. 

After co.pie.i„, PROB. on the last activity of a n^odule- th. instructor 
.directs participants to co.pi^eVheLPG on, he .uestionnai^ 

anv ..uestionabie. or-.not satisfaptory. ra..„fe/ p:ease:be as 
-possii^ie about what you thin, should be changed . " 'pive .inutes „il, be ' 

aiiowed. for this step. .:che PROBZ i,istru.en. 1= then coiiec.ed. ieavin. the I 
PROHES for iho coming modulesln place. / ; - ' 

: PROBE may b. Rev pu„chpd dire=,l^ fro» ie: ,uo=tio„„ai^^^^ 

card columns are indicated next to eabh response box. The number inside , he 
box is punched into the coium.,indJcated. The results of the data anaiysis^re 
dispiayedjn their .final form by a custom-make computer program. 



PROBE General Instructions 



PROBE ,U3St.o„„.,,e nt da.l J.^d U^^^^ ^^^^^ ^ '° '"^ ' ; 

copy of the questionnaire then rpiri th« f^n course. Please scan your 

you understand the kindol inf»n^uon ^tTd;"' *"^""^«^'" ' ^'V w,li help 

* 

THE MEANING OF THE SCALE ' 



1 = 

2 = 

3 = 

4 = 



satistactory. the actjvuy dOGs not require revision ■ 

..t_.e: tho^.^,^^^^ --.en. ...on 

not satlsfaotory. vou strongly recommend that the activity be revised 
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THE ITEMS . . 

n new program and several teachers Th^ ta^^h^ advocating 
. ■ >-'«j. ttecn-iiHib, j,ne teachers were lnst■^M(-^■(^l^ - 

eJther resist or accept tho changd he iidvooated The rls^ o ,hi , 

about to4.pi:^^:S'™0BE'::e^^::':^=^ -er and .ou are 

Hease rate the extent to whiffh- ' i 



(a) 



In general, this item asks, "Is the content of this ac'trvitv- ,nr-i ^■ 
Ideas, concepts, princ^nJes qL-fli« i activity including- 

to or wou?d ;o^u e,=r;fi?h-V^ l^p^Vnt-'to t^e-'aL'r 



(b) 



^^MmmUm.MaMMnteci.so you can^understandjti Was the ' 
^ ir'Tan'f " -Clear and understi^SaHZ^^irner-free of 

complexUios? Remember, content inciiides ideas con- 

Tmi ofT.'r "' ^"'°---n^Ple the content wasThe 

; skill Of Identifying consumor resistance .nd-acceptanco behaviVs! 

- was this skill presented cloarly-or wopldTecommend rovlsion'? 

. - . ..^ _ . . '•-85 ' --. • : ■ ; 



^ waaKnosses f ^U^: f,^^*^^'-^ Procedures to ren,^y 

: : or were thay diarid^^ ''""' ^^^^"^^^^^ ^^^^^"^te^Sarn^^ . 



D-5 



(d) 



you think thS^sho Id h^veS? -Polled out ? Perhaps 

: tion toahniciue.,^^ assignment on observa^ 

Jump into Vrole iJayS^ actStt^ "^d a warm up before they can 

shouJd have been done ah^ari ! ask yourself,, "what 

auccessful ? T time to make this activity more 



Rate alsn ! , - - ■ - ■ . 

' Jhe way thg matgriar^as nrA«»ntrf^ th^ ™ ,% i / ' 

V: ; sumer raslst^^ 

: : ^Did It work?: Were^ther^S - 
Clear? O.^^ 

(S) liSJmfliit^^ the facilities^ ^ 

appropriate and comfortable ■^^V^TThT'f lacilities 
:r Wore you froe of nclse-'S 'dls^l^^"^:;"^^^^"'^^ 

; THE LOG , : ^ - ^ ' 

■t ^what to GhangTwheTWI^riT^^^ scales above do not tell us ■ 

questionable Use ^s^^ T activity Is not satisfactory or - 

; iMsf^ctoiT rat ngs^^"^^ ^^^^^^^ questionable or not sat- 
to change / the 1^0^° 

the:io,:after^eh^^t^^k^ 
• You:may, ihowever/ w sh^t<^ma??^^ modulr -(a set of activities .) 

> you; rate «n activl y ^^e^ "^^ to yourself at the time : 

;v|ou:complete:th^;i 



GENERAL jNSmuC TIONS FOR COMPLETING THE SCmml 

are generally n.ora valid than prolonged r^e^^;:;?. ' ''''' impressions 

^oS^^&nmm^ ^rr ^^-^^-^ 

If you .issad part Of an activi^/or io?t"S 

so.e reason , than don't respo^ to th^ l^^nnSMlt 

not be influencad by you ^^g^ '"^"^ evaluated-^ ^not you . ^our grade will 
lhM^_^ou^ v/hat v^e need 



money can't buy=-we need you! 



I 

a. 



mm 



Difections: Place ypur ratings in the 

boxes undif . the activity 
.Just completed, , . 



SCALEi I s superior V:, ' - "' 
2 - satisfactory . 
: 3 - quistionible;:;: 
4 = not satisfactoiy 



"ACTIViXY 



0 Please rate the extent to which; 



.a^theconciptsM 

^^b, theniaterjal was presented so youtan undefStincl^it. ......... Q^O/^ DC^s 



^cyyou were provided with helpful feedback on your progress,, . . 

i. the time alloted was appropriate' (rM^ : 

Rate also:. 

;e . the way the; material was presented , thi mithods/iiiedia used 

f.: the pality of the instractionil rnaterials . . 

ine quality of the learning environment . , . , ... . . . „ : 



Ml 



HI 



«7i 



LOG; (Explain your negative ratings, B 



egin ycpstateKents With tfie number of tM box you are writing about,) 
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■ INSTRUCTOR PHOBE. 



Directions!. Place your ratings in the. 
boxei under the activity 
just completed. 



HCEEi 1 s superior- • ' 
• • . ^ ^ sitiifastory ' 
3 s gueitionabla 
... . f = not satisfictoiy 

ACTIVITY 



Please rate the extent to which: 



b. participants had the necasiafy entry behaviors.,. 
^Rate also: 

c. the way the material was preiented, the methods/ 







n n 





n 



"'^^'^"^^^ • .-.......„.,...:. 

<i. the appropriateness/adequacy of.the materials,,,, 

e. the appropriateness of the time allotment „„,... 

f. the quality of the learaingeTOronment,, , , . „. „ QQ QQ 



3i 



L'OG: (Explain your negative ratings) 



Module QuesUonnaire 
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OP 



Please rate the extent to 



"^'^'^ .^fe^^odule's activities formed 



ganized whole as opposed to K^fhV^^^^ 

der or purpose:" ^ disjointed or isolated with no sen 

^Superior - ■ 
^^Satisfactory 
Ouestionabla 

^Not Satlsfactofy : 



ell y 

SB of 



Explain I 



Do you think the order In which the 1 
satisfactory? " 

____Satisfactory 
^Quostlonable 
,Not Satisfactory 
lain: 



oarning activities v^/erc presented 



was 



vGeneral Commentsi V""' 




OBSERVATION SCHEDULE INSTRUCTIONS ' '^v ^ ^ " D-10 
DEFINITION AND ECPLANATION OF TERMS • ^ ■ ^ 



Agenda Ttftm 



. ^'''^ '''■^-'-PO-.the instructor's Agenda that I. currently being Ob- 
served . Me,el, . rue the a.e„aa step number you are;observ.„,^ 

Time Auntt^r^ : ^ , ^ : ^ _ 

Writ, the number c, Minutes allotted to this step (as it appears in the 

Instructor's Agenda) . ^ 

Time RennrH 

Record the ti.e (fro. a cloc. or „rist„atc« that the .ronp begins each 
agenda ite™. Then the ti.e actually conJu.ed for each step can be dete^ined 

at a later time. ■ ^ - ■ ^ 

Siibstantivie Input PrnviHorj ? 

Respond "yes., it the instructor provides ^ny substantive input during the 
Observed step. Substantive input refers to any concepts, ideas, principles, ^ 

Skills and the like that relate dlrectlv tn ^H« . . , 

ate directly to the material covered. If the Inatruc- ' 

tor provides none of this kind of input but onlv .n„. • ' 

input cut only engages in managerial-related 

behaviors then respond ..no... If the instructor did nothing „ore than read, 
repeat and clarify instructions, for example, you would respond ..no. .. When- 
ever you respond .'yes.' then note under explanations the nature of the input ' 
•provided V - - - -.. 



Instructor n pntrol Provldflri ? 

; P""=lP^nts so as to help tham accomplish thoirusk. -The Instruotormlght 

Offer onccuraglng or reinforcing remarks when ^klclpants are on the right : 
track. He might paraphrase to facilitate understanding. He n,ight redirecf- 
partlclpants when they drift off task. Respond .yes" whenever the instructor 
provides any control behaviors during the observed step that are not part of 

the instructor's instructions. Then under explanations note the control be- 
havior and, if possible, the Situation that the Instructor Is responding to. If 

no control behaviors occur, respond "no. " 1 



Pb i ec ti yes Met ? ' 

. m about whether they were satisfactorily met, if you th.nl; so, 

respond "yes. . -^^ -et, -r,,^; ..^^ 

- under comments . • : 

Devlatjon.q ? 

/ Respond "norif there are no departures from^the in 
■ :.: "yes" If there are. Generally, other columns on the observaUon schedule are 
| ; intended to catch most departures , such as time; and Instructor Input, so use. 
^ this column as a catch-all for deviations or discrepencies that don'l fit elsewherej 

Instructions m^^^r? , _> 

-- . If it appears that. participants are able to understand and follow con^acily ' 
the instructor's instructions then writa"yes." -If not, write "no" arid rota the-' ^ 
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. : problem under explanations. 

■ ■ Explan at ion s and r;,no„, . . - , . . ' : . . . 

This ooiumn ha. two functions, to ..plain and describe treble spots' 
; Idenuaed In the previous columns, and to record any other noteworthy ob- 
sewatlons. In particular, any environmental conditions that hlndorsd oroduct 

page numbers, missing materials and reversed pages. 

GENERAL INSTRUCTIONS 

Don't fj^H h»h<p-| 

. . It IS important to rema current m your observations so you don't miss 
and have to re„,ember bac. to past steps . If you can't get a^^^ 
. down , make guick notes of the major points and fll, them In later, (leave 

space!, You can use the tape recorder to catch mportant exchanges and conr- 
; ments-talk Into It yourself If you need to get something down fast. 

Observe t he trios whon t hey !;»nar;.>.. \ - 

You may not be able to observe every trln h„t H„ „K 

. vo e^ery ino but do observe some— take the 

tape recorder with you if nos^ihlo r-„„.. 

, y u ,t possible. Continue to note trio departures from time 

■lines and from Instructions; note any difficulties they may have. 

Observe riepartum ^ nm/afgation.q : V 

. Some Of the most important remarks from participants land Instructors, .vlu 
con,e at the^concluslon of the module. Try to tape record and make com„,ent 



In general, your Job is to capture the .ssenca of „^at ,s happening i„ 
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i;to improve them will appreciated 



your task as well as advice on ways- to ir-— - - ^i-^?^ 
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OBSmTION-SCHEDUlH 




Observer's name 
Instruttionai SystinL 



.Modulif 



Explanation and General Comments 



FINAL- QUESTIONNAIRE - 
DIRECTIONS 



ThJs ^stniment is designed to determine the typai of sHlis a 
CDOseri trt riiiHn^ v-nvf '->pi.a oi sKias a Student 



IS exposed to during EPEC, 



contr-.^:L*'.o!^enM: ^^^^^^^^ ^^^^ --^^^"^ '° 'he 

clrciinc, the appropriate numbe" '° competence in that area by 



EPEC 



: Not Apjncable-™ designed to afleot student competence 

In this areas 

2. No Effect--Although EPEC was presumably designed to affect student 

competence in this area. It failed to do so.,™: ' '''^^.^"V^ 

3 . Mintoal Effect--EPEC had some effect on student competence In this 

..... ^TQBL « ^- . ■ . , . 

^. Largo E«ect-EPEC^a^M,hly boncfiol.l In promoting student oompetenoi 

5. Complete Mastery-Asa result of EPEC, students ars fully qualified W 

this erea^ : 



1 2 3 4 5 



FINAL QUESTIONNAIRE " r- 
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■ , ' ' ■ p;<J^s^/""="°" ^= ^ ^''^^^^ --"""tional Change 



o 1 2 3 4 5 ■4 . pan usa some effective group Interaction tools -uch as 

^«^"storming, fishbowl and role playing/ as 

i::.::^: ^,^^. ^^ observation skills. 

occur between. 

; ' ' ' ' ' 7- C^n deveiop^select, and use process evaluationlnstruments, 
: " " ^ \ 8. Ca| analyze and report major results of data selecteH. " : 

1 V'^^ ^ "^^^^^^b^^^^ °f and receiving fei- 

" ' ' ' vation!" ^""^ ^^ affeotivenass of an implemented inno-' 
: 1 2 3 4 5 11. Cairfonnulate a problem statement, 

.^....l„..2^-3 --4-^; 5 12 . Can perform a force field analysis . ' ^ ~ ' ' ' V ^ 

' ' ' ' ^Id anal^^^^^^ '""^^''^^ alternatives from a force 

: 1 2 3 4 5 14,, Can install a specific educational change . 

: 1; 2 3 4 5 > 15. Can explain the difference betvreen Installing an innova- " 

tion and evaluating the installation of an innovation. 

:1 2 3 4 5 16, Can list and describe advocate strategies. ' . ' 

V ; • : ^ ^ ^ ^ '^"^ d«=9ribe organizational characteristics. 

2 3 4 5 18. Can describe the role of the process evaluator." - ^ 
^12 ,3. 4 S ^,19. Can evaluate the process of educational change. . 



172 -3 'A S - 20 . C^njdontify the kinds of dWsions an evaiuator 



„ 3..:.; j = 



services, 

J 2 3 4 5 .„d describe the stngos Of .doptlcnk an iVnovav 

13 3 4 5 24, Can a 

innovation, 

1 1 3 4 5 
1 2 3 4 S 



ssess the level of acceptance* of an 
25., Can assess the stage of adoption of anlnndvatlon. ' 
hLvtrV;'' '^'"^'^ — Pt-oe and resistance 



1.2 3 4 5 P Can recognize consumer acceptance and reslstan..^h 

haviors in human Interaction. resistance be- 

' ' ' ' ' tan^o^tfwf f °^ ^ 

tanco toward an Innovation. . ^"u it,his / 
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TABLE 1 

" module- " su'mmary ' - " - " " " 
Data Indicating Revision is Necessary 





Activity 






Element 




#2 


#3 


^ A 
lr4 




#3 


#7 


#8 


#9 


Module 


Content 






















Objectives 

Environmental 
'Condltidns 

Media 


















- . 




Materials 
Feedback 
Time 










/ 








■ " ■"'■^^^ 




Placement 








i 
1 










/ 


1 * ■»* 



Data Indicate tHat criteria are met: Revision not needed . 
Insufficient Data 
^ Not Applicable • 

irtlcipant Data from PROBE and the Module Questionnaire 

: TABLE A-1 ; 
-i: TABLE A-2 , ; 

TABLE A-3 . : 

'DISPLAY A-1 

structor Data from the Instructor PROBE and Log. 
V .TABLE B-1 

DISPLAY B-l , ^ ' , - 

bserver Data, from the Observation Schedule 



4 TABLE C-1 V 
b-TABLE C-2 . « 
tt DiaPLAY C-1 : 




^3 
I 



TABLE A-r 

. PHOBBSynim'ry! Modu^^ 
Propoftion Chocking Satisfactory or Superior 
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! Variable 




Raiivant 






Environmsntal 


Msdli & 


Mitiriais 










Undirstind 


Feedback 


Conditions 


■ Mathods 




i 
























































4 


















s 


















a 




































a 

/ 9 




































X 



















li 



■■; '■ ■■ DISPLAY' A-i- ■■■■■ ■ ■ 

Participant Opsn-Ended Remarks: Module " 



itivity # 


Variabla 


Participant' 


Remarks 








M 

' CO 

\H 

• • . , _ ■ ' • - .. ■ ■ ■:; ^ 



.as 




TABLE b'^I- 



Instructor PROBE Ratings;, Modid 





'' " . ; V .' ■■' .. ', 


. , ■ , ■ : !■ .J, .rf-'iAV !• 






■ "■ ; ■. v. -J 1 :• ' :' >.-,■' ' ■ ■ - . ^. ■ : il'v .. =' i • ■. . . ""; . 



Activity 



I- 

2 
3 
4 
5 
6 
7 



Content 



10 



Environmintil 
Condition! 




Variable 



Madii &' 
Methods 



Materials 



Iff, 



Time 



/ 



1 ■ '^^ii. 
Behaviors ^ - 



* t ^ . J, - ' 



..■■.."A-.'i-'iiiiifi, 



1%' 



■ I- 
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DISPLAY 

Instructor PROBE LOG Summary : " Module 



Variable 



Remarks 



115 



iiii 



table: c-i' 

Obse'rvation Data; Module 



Activity' 



i Description 



1 

* , \ I . 
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VariaWesNeidihg Attention 




■Objectives 
. Met?. 



'Inatmctions' 
v:Cliir? 



■ Other 
Deyiations 



k mc 





, v.. V.-,. 





.... DISPLAY . 
.Test Site Infofmatlon 




lElESlSLli^Ili^ Date Test St 



age 



if-. ' 




APPENDK P: DATA ANALYSIS INSTRUCTIONS 
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:[ ; : \ DAm analysis instructions ' 

J: SmGE 1: ANALYSIS FO"^ 

Complete the data analysis forms In the follov/ing sequance: 

■ : ' Table A-2: Provide a frequency distribution and cumulative percentage dis- 
— V tributlon of the participant PROBE ratings for each activity / - 

Table A-1: For each variable of each activity provide the cumulative per- 
centage at the satlsfactOiY level as obtained from Table A-^Zl, 

Display A- 1 : List the participants comments from PROBE and the Module 
Questionnaire grouped according to variable and activity , ■ 

Table A-3: Provide a frequency distribution of participant responsjes to the ' r 
; ^ Module Questionnaire.: - : ; 

Table A-5r List each performance test item along with the number of parti-- .: 
i icipant 

. cent of correct responses p : ^ - . . _ . _ ^ 

Table A-4r Provide a frequency riistHhVit Inn of total scores, - 

V Table B of Instructor PROBE ratings for 

each activity/ 

/ Display E-1 : List the comments from the Instructor PROBE grouped by element 
; and activity * ■ V 

Table C-2: Provide the information called for by the column headings (enter " 
^^Y9^ Jot those agenda items needing attention* Under the heading ''time 

match? indicate the. time; discrepancies: f 
^ minutes or /-IS minutes, -^^r ' .^y^-Wr'r r::^- ^ 



Sa^ffl TablevG-l: Condense-the information In Table'C*-2 so as to summarize the 
iii^ljresults by activity , as opposed to an agenda item summary. r 

'^$S r^^^^^^^ C-2: List the observf^r's nommpnfr; nrnnppH hy ^gAnH;^n-^m 



ggST^E^Ir^^ OF ACTIVITIES 



Sii^iiiM&n^^ criterion variaole of interest to the particular element under 




.iVr---'"-': ■ 



Infomatlori Eufees for: the 'Data Analvsis . 



.:±. '. 



Critefion VariiWe 



Decision Riils " v 



.-- Analvsis Fom 



i ■Iv,!.'- 



^21^- Substantive- content q! tHe U ;: .. . , , ' ".; . 
P5^r:lnust;:not;be provided by the Jnstructlonil manager: 
|. - ;:bV.. sh6uld bi perceived by participants as relevant- 
r}2'* Purpose of the LA ; (none) ' ' : 

Perforniflrice-iiased "Oii iectlves^ofitia M "^ . ■ . ? / 
g^^|j:fnust:be:attained.;: ^ -"v' v l-:"-'. : ' 

1^4 Media and Methods of the LA ; ■ ' . - ' ' ' : 
t^ryr^i'inust . 
participants, ^ ; ; f ■ ■> ■ . .;. ., 
'vi,' the instnjctlonal manag : • 
J^i^bi; must be^inipliine.nted as specifiid , ■ ^ ' : ' ; • ■ : 
■If^ -Matefials-Qf the'liA : ,■ ; . 

liS^iraitniiist bi-:perceived;by particlpantsas accipuble* : ; ; 
g;:br.inust be •perceived by th 
li^ii'^^s ■appropriate ' 

Learning Environment SMClflca^ioni of the LAi . " , 
'■y^^avrfnust be |ercelvedj8s implemented by the qbier Vir 
P^^fc^ust bKpercelvad as necesssfy ind sijfficient;by • 
the- observer . -;---'--";"V--" -'^ •.: 
"2, ^;the^ . ■ -^-^^ 'v:^ . 

V ' 3, tiie' participants, ' • ' 
^:4^7 lntry'6ehavlors of^the'LA r;" ' - ' " ^ ■- r~~ 
, 'must be :perceivid"as necessary . sufficient and met 



0'i'',^-;i»!the, observer,' and: '. 
' 2v,the.lns£ructional manager; 



i'--;-. 



'"" " ■■ : i."^' . ^. .. "... .:. .. . . .. ■ ' ' ■. " , " ^'ll^ .-^ . \ .., f V . , , 

^ f provided content=9- ^ >,^TatieC*lSi^^ 
: :pOiitiye'responsesa5l ' / 'Tablei'-l .iii. lSsplay^ 

;J Per Cent (xi90p90l : - ■ ■ ; :Tabia^ " 

affirmative ;risponi;^8Si: v ^ : - Table A-l-; Displ^^*!-'^ 
' effinnative responie - v;^ a ■ Tabie^B^l pisplay;B-|^v| 
f devlations-Q ■ ; "\ ' , Table 0-2 Display C-1 

;h affirmative responsi^85fe:;|?:'M 
. affirmative reiponse: ; : : . Table B-l /Display B-1 I;.; 
f departures-O.^ ,,,.,"::, ' Display C-1 

:f reporkd::problems-0: . 7. :. -..-y : : ;;: ..; . Display C-l:^# 
' :;iffifmative respoiise; v. ; ' ;MTable l-i; , Display B-l^^^ 
vaffirmativl^^riSpons^8S%;ii;^^^ 

" ,1-—'. ^ ' ' - J .. p ,1,*-. - 

f reportadi:prQblem=0 -::*^;iTable;C-2j,(ttsp^^ 
afflrmativeiespHie'v;-- ' DisplayB-i^^ 



f .:*See test;critsrla for a breakdown of this category, 
lERJclr'' 




mil D (cont.) 



Cfmim Variable 



Deciiion Rul§ 



Analysis Form 



, 8. Tim^ allotment of the Ik ' 
v"' .a. must coffespondwith'thfi^aetual timt consumed - 

b. must be perceivid as ippropriite by ; ^ , 
-^ 1:: -- 1 . participantl, ind 

' 2. jistructlonal manager . 

9, ff^Hh^^^k PrnnfifiHrftg of the LA: 
V a. must be pirceived by participants as adequati 
^^;^''7"inrribt'dlsfuptW " ' 

;iQ. Plarpmpnt of the Ik 

' a . must be perceived by participants as eppfopriati 
in terms of sequence and continuity . 



f discrepanciei-O 

iffirmativi rfSponseSBSI 
^affirmative" response— 



Table C-2 Display C-V-;# 



TabieA-l . Displayi-l^A.^^ 
..TableJdrjiiplay:B-L;:.i| 



affirmitiveresponsi,;S5%^ 



affirmative respoi 



i j. t!i'' '.'s\ - ■]".«', 



4" 
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. ExamEie: For element 1, "Substantive Content of the LA" the first criterion 
variable is "must not be provided by the instructional manager. " 

B. Locate the relevant data from the analysis^forms for the criterion variable 
being examined. (Table D lists the analysis forms containing the data appro- 
priate for each criterion variable) ^ _ . ^ - ^ . 

Example: For the criterion variable selected Table D refers the evaluator to - . 
Table C-1 and Display C-1. Here ha will find data pertaining to the first '-^^^ 
^criterion, variable.^. _ „ _ - ' - * 



C, Review the data available and determine , whether the criterion Is^met ; 
T according to the decision rule specified in Table b • - ^ 

Example ; An examination o£ Table C-1 will reveal either a "yes" or a "ino" - 
.;. under the heading "substantive content provided.'? *" ' . . . ^ ■ " , 

D. If the criterion was met, proceed to the next criterion variable of the 
element being examined and repeat steps A through C . When examination " 
of a particular element is completed and all criteria were met, thin enter a 
"+" in the appropriate cell of Tab'le 1, the module summarsN 

Example: Under the heading, "substantive content provided?" a "no," has bean 
entered. The decision rule in Table D states that the criterion is met if the ,' 
frequency of substantive.contentrprovided-during-the-activity~ls-zero.^ Since / 
no substantive content was provided the criterion is met. Therefore, the 
evaluator proceeds to the second criterion variable under the element "sub- 
stantive content of the LA," which is, "should be perceived by participants 
as relevant, " and repeats steps A, B and C. If this criterion variable is also 
■ satisfactory the evaluator enters a "+" in Table 1 of the module report under 
the column of the activity being considered and In the row headed "content." 
He then proceeds to the next element "Performance-based objectives of the* 
.LA," and repeats steps A through D again. . 

. E. If during step C, the criterion was not met, then instead of a "+", enter 

the appropriate code number in Table 1. The code number represents the analysis 
^_£°f"^J^^^ contains the information suggesting revision is needed. 'The key to the . 
_- code cs displayed at the bottom of Table 1. Providing this coded entry in Table 1 

allows someone reading the module report to locate the source of any data sug- 
'4 gesting a need to revise an activity. 

. Example; If, under the column of Table C-i headed "substantive content 

provided?" the evaluator found a "yes" for the activity under consideration, 
";%then he would enter in Table 1 a "7", the code for Table C-U 

^ not pos'slble to determine from the data obtained whether the criterion 
"is met, then a ".-" should be entered in the^appropriate cell of Table 1 indicating 
insufficient data . 131 ' 
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There may also be times when a particulaf element does not apply to the 

. activity under consideration. In this event, an N/A should be entared 
In Table 1, 

- G . Whenever a crltorion is hot met and there is evidence suggesting revision 

- that evidence should be discussed in the text of the module report. This' 
discussion should summarize and evaluate the strength of the evidence for 
revision.^^n effort should be made to relate other obtained Information to " 

. the.case being .considered in order to better understand the nature of the pToblem 
The evaluator might listen to the tape recording of the activity, for example 
to get a better idea as to what is not satisfactory. .He should also discuss any 
open-ended remarks that relate to the problem. If there are arguments against 
revision of the activity, these also should be considered. An effort should be 
made to explain what is causing the problem whenever this has implications 
for how the activity might be Improved. 

H. The final step Is to recommend how the activity might be revised to 
strengthen it or alleviate the problem, or even to recommend that the evidence 
Is not sufficient to warrant revision. 



MODULE REPORT CONTENTS 

The report should bogin with the following opening paragraphs! f 

The results of this module are lummarl^ad In Table 1* Each 
activity is considered in terms of its component characteristics - 
or elements^ that Is / the activities content/ objectives, - 
.materials # etc* A + indicates that no evidence was obtained 
to suggest' the element of the specified activity needed to be 
revised. One or more digits indicates that there is evidence 
. suggesting the activity be revised . The digits are codes that 
refer the interested reader to data summaries and comments' in 
vr. Appendix A that constitute the evidence for revision. The code> 
and instructions for Its use, also appear in Appendix A. 

. In the following discussion we will dispense with conslderat ion 
- of activities judged OK and consider the evidence for and against 
revision orthe remaining activities , ^ 

Table 1 follows the above paragraphs . The next section of the report is the 
Discussion and Recommendations * This section should be organised by learning 
activity, and within learning activities it should be organized by element*/ 



The last section of the report is a summary of the recommendations. The 
.arfelysis forms-are included as Appendix A and the raw data is included as 
Appendix . - - ■ ' 




*||*^^^ ^ *1* ~ i « J'-i^i V «is:^>% 4.|4J.« * 

* 'T-^f- *^jr -^T^ ^--^--^ * -i*-^ ^-t--* ^=** * ^ * T ^ -» # ^ * 
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Incroductlon 

tharaeter tstlcally, an experiment l5i%*olv^8 a colleGtlon of treatment . 
c nd 1 r 1 c r. fx < 1 . i* * t r e5 cn c 1 c v<j 1 s or t r aa t men t c omb iiia t Ions ) , a 
col lection of rxt^erimnta:! units » and an explicit plan for assigning 
treati^^nt conditions to units,; ^Jor purposes of expssitlon, we can 
divide exparlMnte into chose in which tima plays an Important role and 
thoB-a in ^hich it 4oi*8 Hot, Time My enter Into the eKperimenCal plan 
in r*ev«*ral vayR, For exa^ie^ 1) at some point during a aequenee of 
rcpexiced measurements of the eKperimental unit a treatment condition 

be Introduced, as in trend analysiSs 2) the experimental material 
"^ly be succesalvely axposed to saveral pre^specifled treatment conditlone 
nx%6 mcfw^suxed after each, as wher^ assessment of order or residual effects 
Is of Interest^ 3) ^r^atment conditions may be administered to experimental 
units ov^r timii in aych a way that previous treatment conditions and 
responseB to the^j are used in determining the treatment conditions which 
follow* Note that in eKamples two and three creatmant coodltlons are 
administered over time* But in the second example the exact treatment 
condit ions are determined a priori , while 1r e^asiple three, they are 
determlnad during ,.tha exp4riment as a function of accumulating data* 
For convenJLence, we label the three examples as Instances of repeat ed 
measur emejit i serial > and sequential designs^ resp^ctlvelyi 

The present research is concerned ?i^lth gequentlal expei^lmer.tatlon. 
Experimental deJignl which are aequentliil In nature require that the 
experimenter consider both how the ensuing treatment conditions will be 
changed or adjusted and how the process will be discontinued , 



i.e. a "stopping rule". Sequential experimenta can be differentiated 
from one another by considering whether a forml or infomal proctfdura 
is used when adjusting treatment conditions, whether tha stopping rule 
is formal or informal ^ whether or not more than one factor le usad 
(l*e, multlfactor experiment employing several differen\t treatments)* 
whether the independent variable or dependent variable le continuous 
or discrete, and by ctmslderlng the purpose of the procedureii {e.g. 
locating maxima). (Far: ;a general review and bibliography of recent 
work on experimental design * Including the topics dealt with here* see 
HerEbarg and Cox (1969)* For a current review of ^ the design of sequential 
experiments, see Chemoff (1975). Wetherlll (197S) provides a useful 
Introduction to the subject of our paper)*) 

Examples of applications of sequential designs are not plentiful 
in the educational research literature. Meyer (1963) preaeata an 
application of response surface methodology. This methodology la seen 
as sequential in nature by Chernoff and by Wetherlll/ Response surface 
designs are factorial in nature, employing several quantitative Indepen^ 
dent variables. The dependent vsrlable is often assi^ed continuous 
and a polynomial function of the Independent variables. Purposes of 
these designs include locating maxima or estimating parameters of the 
polynomial. Decision rules which specify the "design points" to use 
in the next stage and when to stop the process tend to be Informal. 

In contrast to response surface methodology are stochastic 
approximation techniques in which a single continuous independent 
variable is Investigated and where values of that Independent variable 
are determined formally as a function both of the preceding values and 
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the reBponses that were obtained when they were administered. A techniqu 
due to Robbins and Monro (19S1) is an example. It® purpose is to find 
that value of the independent variable, say 9j such that the expected 
value of the depenilent variable given 8 is equal to some predetermined 
constant* 

Our research inve^tigaHes two examples of the Robbins-^Honro 
process and three va^lartt procedures which were motivated by it* Much 
of the previous research in this area has been focused on asymptotic 
properties* Chemoff (1975) gives a brief and readable review of this 
work. Of particular Intereet here is a paper by Hodges and Lehmann 
(1956) because it suggests assuming a linear relationship between the 

independent and dependent variables and also assumes that the slope 

- ^ .... 

parameter is knoTO* While these two conditions would seldom be met in 
practice, their theoretical and numerical results provide a basia of 
comparison for empirical findings* ■ 

The Robbins-^Monro procedure has been modified by some researchers 
so that two values of the independent variable are employed at each 
step (e.g. see Venter (1967)). This procedure has certain advantag^Si 
but only the case in which a single value of the independent variable 
is used at each step is studied here. 

Presently, we know of no application of Robbins-Monro procedures 
in an educational experiment. However, the technique has been applied 
to a measurement problem by Lord (1971a, 1971b) . Those two papers 
dealt with quanta! responses, a subject not dealt with here* (for 
this reason and because we did not want to define the values of the 
independent variable a priori we have not considered the "Up and Down" 
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methodl of stoehaaCle approKimatlon.) The present aras of invaitigation 
has similarities with sequential eatiMtlon, but alio iOM 
differences* For the eatimatlon p^obl^ only the "stopping rule" 
need be considered* for no independent variable le manipulated* 
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. - - The Problem 
Assume that the eKpfirimenter ■ a goal is that the value, of . a 
particular population mean li to be changed from Its present valuer 
6| to a different value^ a. For example, a population of adults may on 
the average score 5 ^100 on a particular atandardlged reading test 
and the goal ia to Increase that average to a ^ 116 i The eKperimenter 
has in mind a treatment variable (say * nimber of ^ hours of individual 
tutoring) which he knows can affect the average reading score, but the 
exact nature of the relationship between readitlg score and tutoring is 
untaiown. In other words , the "end" is taiowni but not the specific 
"means", and therefore, the appropriate value of the independent variable, 
or treatment condition, must be found* Mora formally, the eKpected 
value of the reading score Is a function of the independent variable, 
E(y) ^ f(x), and the eKperimenter wishes to dataraliie that specific value 
of the independent. VMiable, 9, for which l(y) a/ or l(y(3C * 8)) ^ ot. 
For present purposes It is assumed that if k > 0 then ECy(x)) > a, and If 
K < 0 then ECy(K)) < u. Given this situation the eKpsrlffienter Can select 
an initial value and thereafter choose the value of the independent 

variable as h ' , ^ x ^ a (y^i^y - a)* The a^ : are selectW to have: 
n + 1 n n n. n n 

several characteristlcas the most intuitively ifipor tint of which is that 

a ^0 as n— "at a suitable rate". One possible definition is a > 
n n n 

If appropriate a are chosen, such as i, Robblns and Monro (1951) proved 
n u 

n^«, X — The experimenter, of course, must have some feel for the 
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speed of Gonvergencei and how this canvergence affected by the 
choice of x^, the relationship between E(y<x)) and and the density 

of y(K). He also must have some idea of whan to stop the experimentation. 
Most of the results in the literature to dates howeveri are as^ptotie 
in naturep with relatively little work being done on stopping rules 
(Chemoff (1975) offers no oitationsg but see Farrell (1962)). The 
literature, as it appears to us» provides little If any praetieal 
guidance for the experimenter . j.- 



9 
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Methods 

Initial results were obtained with an Interactive empirical approach 
using computer simulation teclmlques on a tlme^shared CDC Cyber 74. 
Many CDmpu Car runs were made as the researchers sought to understand 
the importance of the numerotas parameters which can be considered* Following 
this first phase of computer runss during which all the values produced 
from a single sequential experiment were often observed^ more traditional 
Monte Carlo experiments were performed, replicating the escperlments a 
number of times to obtain estii^tes of how the procedures operate 
"in the long run"* In euimaryi the apprQaoh used combined both an Inter^ 
active search during which the researchers observed the behavior of 
various functional relationships during a single replication and more 
traditional "flKed" type of eKperlments in which a number of replications 
of an experimental situation were made to obtain stable estimators. 

All pseudo random numbers were obtained from either NORMAL or 
MN3F which are a normal (N) random nmnber generator and a uniform (U) 
random number generator 1 respectively. One thousand random numbers 
were generated per each call of these routines and following generation 
they were ii^edlately permuted by an Independent randomlEatlon proeedure 
using the program PEMJTE* All routines are maintained by the 
University of Minnesota Computer Center, 
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■ Design 
The model for the random variable y was y y + where 

U,/^ ^ ECy(x)) ^ f(K) ^ S^x° + + S^x^ + 6^x^ and where e Is 

2 2 

independantly identically distributed either as N(0,o ) or UCOjO )* 

y«x , y«3C 

Following the interaetive search in which many parMietrie ipeoif ications 
and stopping rules were studied, certain cholcei of parameters and 
rules ware made for the more standard type of Monte Carlo invastigatlon. 
These included i 

1 1' 1 ^ 

1. Four definitions of a . They ware a ^ ~, where 
. _n n n ngg n 

Bg la the first derivative of f (x) evaluated at 6, B is the usual slope 

k ik 1 

estimator, and s ^ \l ^ . | , k ^ INTEa/sc], where c >0 and even, 

1 - j 

j ^ max(l, n - c + 1) and z ^ 1 if y(x ) < a, or z s if y(x ) > a* 

n ■ ■ n n - n 

(INT maana "integer part of.") For the procedure employing 6, 

1 1 k 

a ^ - for n < 20 and — otherwlae. In the definition a ^— 
n n n n 

nS . 

a "finite memory" is introduced Into the approximation process i and ^ 
successively positive or negative values 7^(^^) ^ cause larger 

adjustments X , - than is the ease with the other definitions. Both 
' " n + J- 

s and S are random variables and this results in a variant of the 

Robbins-Monro procedure in that it assumes the a to be "a fixed seauence 

.... ... ... " " :n._, ■ " _ ... ^ 



of positive constants;" \, 



2 . Two stopping rules * They were 
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Rl: Stop If n > 20 and If a contained In y + g I - 4-^ ^ ^ 

(where u ^ 6 + Sx ) or 200* 
y.K n — 

R2i Stop If n > 20 and , conaiderlng the last 20 values of z, if 

Ee ^ 9| 10, or llj and the number of "runs" is 9, 10* 11, 12i ©r 13, or . 

if n ^ 200- 

3/ Three sets* i B^i g^. They were [100, .14142, 0, 0] 

[100, .34641, 0, 0], and [100, ,12686, t00S8512, >^, 000023767] / 

. . 2 _ ... . 

4. Two eondition al v ar ianees . They were o = 100 and 25. 

Most "final" e^cperiments were based on 500 replloatlonsi^ Baaed on these 
replleations, the mean and variance were eomputed for (k^ 8) at 

n ^ 30, 50, 100 steps and for both rules, Rl and R2* Additionally, for 
both rules the mean and variance of the number of stepe needed to stop ^ 
were also eomputed* 

a m 1 was Included in the eKperiment because it was suggested in 

n n _ . ■■ ' - 

Robbins and Monro's original paper. Hodges and Lehmann provide results 
on a ^ ""4^ when the regression Is in fact linear, and it has certain 

optimal characterlstioe and therefore was Included as a basis for 
comparisons. In discussing the preceding work, Chemoff (1975) remarked 

that "In the stochastic approKlmat ion case using sequences a^ « £, there 
is no prior knowledge of e to insure that c = B ^. However, as data 
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aceumulate one would hopefully obtain a. satlsfwcory eetlaate of B : ^ 

2 

providing^ the auedesslve are not too elose to eaeh other • 70) • " 

Wa interpreted chase cosmants to mean that when one has V'iufflclant" 
Informatlovr^ne would estimate using the laaat squares estimator > 

0. Initial rasults deronatrated that the instahility of 8 for amall n 
'"caused erratie adjustmeatB and poor eonvergence* * This lead to the 

1 1 - 

proeadure a * - for n - 1, * . 19 and ^ thereaf ter* The a^ ~ : - 

Jl^. . . .. . .. . ............ ... 

procedure was developed during the InEeraetiva part of the present research. 
It seemed raasonable to specify an adjustment procedure which would make 
larger adjustments if E(y^(x^)) ^ a vara judged to ha large. Considering 

.only the sign of F^Ck^) and taking c « 4p for the patterns (+ + + +) or 

. ■ ■ ■ . ' . .. 

(= - - 8 ^ 16,. for patterns like (+. ^ + + - 

and for patterns with two pluses and two minuses, a -^1. This type of 
adjustment assumes that the error distributions are syme trie so that 

the probability of a plus at « 0 is |* During the interactive phases 

of this research* c 4 and c s 10 ware forad to work well. 

Stopping rule Rl employs the standard confidence interval for 

estimating u . This seemed a reasonable approach to consider, especially 

■ •■ " . ^ - y.x ... 

when a ^ ^ is employed. The confidence coefficient, p, used was ,60. This 

n y - V---: ;^ - ■■■ 

value was chosen during Jbhe interactive phase on the basis of performance* . . 
Stopping rule R2 comes from reasoning T -i • ^ 

developljQg the s procedure. At — 8, for syroaetric error dlstribucilons 



-the sign ot y.(% ) * a vsuld be in4Bfm%im%tlr 41.strlbyt»4 mm n l^rt^-^jlli 
variable with psTameter E2 enieatijBlly 3 ^11^ hirf^tli^a^a,! 





.. i^*mits -th% Tim^''^rn ptmmt^ intJki^Xmtr 1^8. Tbm im^n 

of tfe# felsn C^^,> 6)t report^, for tb^ to^lti^iMi . stt^t^ «.».. 

1.2. ATirr*^^ i^qu«r##i biis, (x 6) Is no E reported, but it hm. 



w-Mmitf .^ht^if^^ by aquArlag the av«r#g# biAs and Adding thl« £o tb^ 

" • ' ^ - ^ " ■ 2 " ■• ■ ' 

n n n 

tf^'-w^thGd m^lQrin^ -^ yrmw. Buprnf iot to the other 2i€th©ds, but sise^ 



IlitT ii«^^^#i5clmiirk.M'\ It i« clear frOT tbe rMults thftt : gmerally t 



^^^m priKt^'aT^ lE^ * I iyt4 potest perfersance,. In sltuaCioas vhera. 

lltllf^ mt ^ pridr |r.fet^tioii i» available s^ut the Telatlonshlp betva^ 

i&Smpm^iint d#p^;^a4it v^rlmbles, the rasulcs v©uld lead ua to . 

'- ' : . ..k " ■ \- ' ■ . . ■ . . . . - - - . . : 

uii€ a * ™ Vith c * ^* Ttii3 t^ic^ly does as well as or batter than 



th^ QthMT pr^rdurcn imt ^^loyinR B^^ It Is «rkcdly better when there 

is a wak relatiMfhtp ^letveen the Indepesdrat and dependent variables 
BvA 4 poor ^tsrt is f^jsde (see eoltiras 1 and 2 of Tables 1""7) - In an 

attest to determine the behavior of a * — when a ^ good start" Is 

n n _. _ . 

si^4e» the e3:pcrls«.ts reported In Table 8 with « 6 were carrl^ out. 
We believe the procedure did reagonibly well under these clreimatances. 
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If Sor. l.fon«clo„ 1. .v,ll.u.. ^„j^,,^^^ ^^^^^ 

r,.p.= . « .„d „™b„ „f etep, stop. The^ ,130 ,pp.„, Z " 

"arters ln . f» Here we =i? only .eco^end thai „„e „ta 
•."t 8„e,s about coodltlona a„d „ae that stopping .„l, wy,, ^^i,",/ 



bast 



16 



ERIC 



Educational Signlfleanee ... ■ .r-z-ir _ ■ 

One potential area of application for stoehastie approxij&aeiDQ 
is that of formative evaluation. Stochastic approKiaation can suggest 
values of the independent varlahle which would attain prograBmatle. 
goals, and this infoT.-matlon could be fed to persona dlractly iavolvad.^ 
in program developments Within t framework develDjfed hy Sanders and 
Cunnlnghani (1974) , stochastic approximation coi^d prov:f.de "external 
InfotTOtlon/* for '■fonaatlve Intarto eval a 
simaatlva evaluation is planned, perhaps using one of th^ more standard 
experimental designs i deaign points can be chosen in 1Ch<i region suggested 
through sequential^ experimentations thereby increasing the likelihood 

that the progrm will demonstrate its effectiveness. | . 

I 

In general, stochastic approximation would appear to be a useful 

tectmique In any area where individuals have a goal firmly in mind but 

- - . i - 

lack sufficient faiowledge of the independent variable to design an i : 
efficient, more tradltTBSr experiment. Education Is goal prlented, 
and information about how to achieve a goal is often more important than* 
say, information about the .exact nature of the relationship' between an 
independent and dependent variable. Stochastic approx^mtion can provide 
useful infoTmation about an independent variables even when Its defined 
over a broad range of values, while requiring relatively few subjects 
for Its implementation, ^ _ _ " _ 



Table 1 



Mean and Variance of the Bias (x - 9) at 30 Stapi Whare « 4 



{100, .l4l42,0„0] 



[100,.34S41,0,0S 



[100,. 12686,. 0058512, 

-.0000237671 



,--;a- 
-'• n 


2 

a 

y.x 


c-H(0,o^ ) 
y.x 


y.iE 


' y.x 


y.x 


e-W(0,o^ ) 
y.x 


y.x 


1 


100 


-.'49 
160.93 


.38 
177,75 


1 -.13 
■ 27.13 


.20 
29.84 


-.96 
12.85 


-.79 
14.54 


nB- 


25 


-.19 
40.85 


.24 
44.77 


-.08 
6,81 


■ .10 
7.46 


-.53 
i 3.39 


-.48 
3.81 


1 


100 


-61.16 
72.00 


-61.11 

81.11 


-9.73 
27. 65 / 


-9.25-"- 
32.02 


-6.57 

-„-.20.82 


-6.68 

..^.-,22.91 


n 


25 


-61.27 
18.89 


-60.96 ^ 

20.48 
|_ 


-9.46 
7.69 


-9.62 
7.62 


-6.48 
4.74 


-6.36 
5.59 


1 


- 100 


' -44.86 
1436.76 


: -41.86 
13B5.58 


-5.65 
140.41 


-4.55 
180.92 


-3.14 
108.84 


-2.36 
188.85 


• nB 


■ ;: : -r25;7- 


-36.23 
__„490.86 


-35.41 
577.45 


-5.72 
65.93 


^ -5.57 
51.08 


. -3. 07 
_ 43.88 


-3.10 ^ 
50.68- 






-1.92 


-3.58 


.42 


' .02 


-.18 ; 




100 

k 

25 


298.62 


452.01 


72.73 


97.26 


44.55 ^ 


61.85- 


j -.88 

,■71.44 

L 


-.98 
96.15 


.44 

18.46- 


.19 
22.93 


.18 
10.77 


— .32 . i 
14.73':. 


k 

— Cc= 
n 


100 
=10) 


-10.85 

835.47 

- 1 


-17.03 
764.57 


-.81 
110.47 - 


-1.93 
87.05 


-.06 
47.69 


-.89 

' 49.23 " 


25 


I .37 
378.17 


-1.98 
419.80 ' 


-.67 
24.05 


-.75 
25.86 ■ 


-.10 
24.08 


.26 
11.04 - 



For Tables 1-8 the upper nimber In each eell Is the mean and the lower number 
±8 the variance, T ^ 
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Table 2 



Mean and Variance of the Biasl (x^ - e) at 50 Stepc Where ^ 4 



n 



1 

■ * 

n0 



100 



25 



100 



25 



100 



25 



100 



s - 



-4) 



25 



k 



.100 

«10) 

25 



[100,. 14142, 0,01 



&-N(0,o^ ) c-U(0,o2 J 
y.x - / y.x 



.30 

.93.84: 



.18 
23.66 



-56.87 
62.61 



-56.92 
.16.49 



-20.43 
1036.14 



-20.56 
231.29 



-1.48 
157.02 



-.46 
40.04 



-7.93 
476.52 



. . .33 
169.92 




•:;^¥'^1'^2'^3^ ' 
[100,. 34641, 0,0] 



C"N(0,o2 ) e^uCO.o^ ) 
y.x y.x 



I100,.12686,.0058512-, 
-.000023767] 



,-"u3 



.34 

109.69 



-56.83 
69.82 



-56.61 
-17.82- 



-22.29 
994.74 



^20.91 
236.72 



-1.99 

262.19 



-.47 
58.47 



-12.07 
513 r4l 



-1.81 
229.20 



.17 
15.75 



.20 
27.54 



.08 

3.94 



-8.02 
20.59 



-7.92 

-5i-44: 



-1.38 
158.51 



-2.97 
25.52 



* .07 
39.42 



.27 

9.83 



-.38 
54.41 



12'. 92 



.16 
18.35 



19 



.08. 
4. SSL 



-7.69 
23.88 



-8.04 
5.88 



-1.51 
104.35 



-3.20 
18.06 



-.06 

50.39 



.11 

13.56 



-1.36 
46.98 ' 



-.02 
"12.72 



-,43 
7.21 



-. 28 
1.96 



-5.04 

13.69 



-4.98 
3.12 



-.63 
33.50 



-1.47 

12.84 



.26 
25.33 



.04 
6.11 



-.12 
^2.12 



.20 
6v35 
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8.67' 



-.25 



2.25"' 



-5.17 
15.49 
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3.67 



f 
I 



.-.67 
67.36 



-1.43 

13'. 58 



-.34 
33.99 



m 



M 



.07, 
8.22 



-.31 
22.28 

- .i6 

- 6.72 



Table 3 



Mean 



and VarianGa of the Blag (k * 0) at 100 Stapa Inhere « 4 



[100,. 14142, 0,03 



[100,. 34641, 0,0] 



[100, .12686, .0058512, 
000023757 1 



. 2 
a a 
n_.„ y.x 


€-N(0^g2 ) 
y.3c 

1- ■- - ^ 


e«U,C0,.o ) 
y.x 


y.x 


e-UCOjO^ ) 
' y.x 


E^N(0,a^ ) 
y.x 
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1 
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. uo 
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8.14 


. . 01 ^ 
^ 9 . 75 
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nfl' 
B 

25 
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m 
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- 14.63 


. OB 
2-03 


. 00 
2.44 


.95 


-.16i, \ 
, 1.08 


... . ..... ... - ... . 

100 

1 


53.31 


^ 5r.95 


■^Oa 

13.19 
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15.68 


7.21 


8.42 


25 


J 13.64 


14.56 


3.50 _ 


~ 3.83. 


-.'^ Aft 
1.71 


- 2.07.; 1 
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1 


-8.62 
333.15 
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44.44 


34.35 


8.90 
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25 


-10.44 
74.54 


-10.67 
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-1.39 
7.19 
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-.50 
3.08 


3.3o4 1 














— =S 
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78.34 
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27.90 
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15.02 '1 
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6.84 
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k 
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Table 4 



Mean and Variance of the Bias (* - e) When 

Stopped with Parametric Rule (Rl) Where ^ 4 



[ibo,. 14142, 0,0] ■ • [100,.34641;0.0] ~ [100, .12686, .00585l5;;._*': 

-.000023767] 
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5.93 
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352.92 


-4.32 . 
460.81 . 


.06 
59.17 


.41.1. 
67.44 


-.80 
28.71 


-1.15 
32.94 


-.89 
■ 91.96 


-1.20 
110.92 . 


.10 
14.93 


.22 
15.90 


-.32 
7.28 _ 


-.65 
7.61 


o..' 100 
.,|-Cc=10) 
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' -1.88 
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-.48 
14.53 


-.64 
17.57- 


.42 
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8.20 



The estimates in these cells are based on 100 replications Instead of 500. 



Mean and Varianee of the Number of Steps 
When Stopped with Parametric Rule (Rl) Where x^^ 



[100,. 14142, 0,0] [100,. 34^641. 0,0] [100, .12686, *0058512, / 

- -,000023767] ^ 

t ~ 1 . 



^ £«N(0,g^ ) "e-UCO,o^ ) e-NCO,0^ ) e-UCO,^^ ) e-^(0,g^ ) e-'UCO.af -^j' 

y.x . y-^ y*^ y-^ y»^ y*^ y* : 



100 

1 

25 


20.18 
2.37 


20.21 
1.79 


20.13 
1.30 


20.18 
1.S4 


23.13 
338.^95 


22.23j: 
194.61 


20.17 
2.07 


20.13 
1.08 


20.. 18 
.2.33 


20.14 
1.09 


26.28 
622.57 


24.96" 
464.60 


100 

1 . . 

n . 

./'--" 25,: 


196.44 
627.30 


198.24 
309.76 


87.64 
7087.82 


85.30 
7003.51 


92.26 
7320.44 


95.61 
7372.14 


200.00 

* 

0.00 


200.00 

* 

0.00 


191.49 

* 

1393.06 


196.05 

* 

572.69 


191.22 
1416.44 , 


' 194.62 
622.76 


V; 100 

■ -nS- v:: '-^^ .:: --;-■ 
v- ■;.^;-..s;;vr^\25;;y:^ 


89.66 
5019.12 


93.18 
5523.06 . 


28.76 
788.86 


- 30.11 
1132.37 


28.78 
695.83 


29.03^ 
716.531, 


177.87 
2849.49 ' 


178.80 
2840.74 


76.85 
2214.93 


82.71 
2414.49 


78.21 
1589.28 




80.39 
1468.92 


100 

■ k -^-;/ :;^";-:;-- 
— Cc-4) 
n 

25 


24.86 . 
„ 338.35 


27.53 
610.14 


25.68 
■ . 210.39- 


26.46 
262.41 


26.02 
165.28 


27.16^' 
228. 26f 


25.03 
477.47 


27.37 
711.50 


25.50 
155.27 ' 


26.75 
213,57 


25.87 
137.34 




28.97- 
405.82 ? 


100 

5-(e.lO) 

'25 


53.20 
3807.48 


65.79 
5113.46: 


24.32 
399.95 


25.61 
492.67 


2 


23.96 
50.35 ■ 


26.59 . 
,654.13- 


83.42 
6015.15 


111.00 

6715.59 


27.24 
_ 775.31 


29.36 
1136.49 1 


^ 27.01 
777.34 


26.12/.- 
646.42 



^■^^i^-i-"^^' ^ffr±m^i-i=^p ■\n these cells are based on 100 replications instead of 500. 



n 



Table 6 

Mean and Variance of . the Bias (x^ t 6) Wh^ 
Stopped with Nonpar ame trie Rule (R2) Wiere ^ 4 



[100, .12686,. 0058512, ir 
-.000023767] ■{::, 



y.x 



[100,. 14142, 0,0] - [100, .34641,0.0] 

C-NCO.o^^^) £^1J(0,O^ J =-'"^°'*'y.x' ="'^^°'°y.x' ^"N^°'°y.X^ ^"""^^'^y.X^-; 



' 100 

1 

25 


-.21 
173.52 


.05 
179.81 


-.01 
29.40 


.07 

29.87 


■, -.94 / 
12.59 


-.80--'- 
14.97 


.00 

44; 00 


.08 
44.88 


.00 
7.34 


.03 
7.48 


-.44 . 
3.42 


-.54 , 
4.09 


100 

1 

n 

25 


-48.05 
42.17 


-50.04 
51.09 


-7.75 
16.73 


-8.02 
. 25.63 


-4.68 
J 11.06^ - 


-5.52 

17.34.i 


-46.74 
11.18 


-46.54 
12.08 ■ 


-6.17 
3.10 


-7.01 
4.97 


-3.66 
1.75 


-4.10 
2. 90 


100 


-9.77 

526.20 


-14.57 
631.61 


-2.86 
54.60 


-3.25 
84.38 


-1.55 
18.48 


-2.16,'. 
30.35 


nB 


-9.70 

56.80 


-11.74 
82.45 


-2.53 
8.40 


-3.42 
11.26_. 


-1.48 
4.10 


-1.83 
5.39 


100 

25 


-.46 
.24a. 88 


-2.30 
■453.42 


.53 
77.44 


-.20 
106.10 


-.47 
46.65 


-.60 

65.27, 

-■.„» , 


-.04 
61.75 


-.50 
92.97 


.49 
16.49 


.07 

25.41 


.12 
10.77 


l4.-87':f ■; 


100 

k 

— (c=10) 


-8.11 
391.91 


-12.36 
502.21 


-.34 

42.68 


-1.73 
64.64 


.29 
y-30.43 - 


-.25^: 

- :".33.71 ' 


n 

25 


.49 

138.40 


-1.48 
215.96 


.00 

10.91 


-.12 "~ 
i6.l9 


.61 

6.55 


.52 
7.37. ) 
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Table 7 " . 

Mean and Variance of the Kumber of Steps 

When Stopped with Nonpatametrlc Rule (R2) Where x^'"* 4 



^^Q' ^1' ^2* ^3^ 



[100,.l4l42,OjO] 



[100,-34641,0,0] 



[100, .12686, .00.-i35l2, - .U 
-.000023767] 



1 G- e~N(0,o- ) E-UCO.o^ ) E-N(0,o- ) e-U(0,o^ ) c-NCO.o^ ) e-U(C,of )'< 

n y.x y.x y.x ^ ' y.x y.x ' y.x' ^ y-X':- 



100 . 

1 


32.42 

226.32 


31.79 

187.77 


32.40 

223.18 


31.86 
187.39 


35.06 
294.07 


33.45 ~ 
247.42 


32.60 
224.71 


31.98 
187.41 


.32.60 
224. 71 _ . 


31,98 . 
. 187.41 


34.81 
240.51 


33.31 
237.54 


100 

1 

n 


173.71 
2234.70 


136.65 
3301.21 


61.13 
1369.02 


50.38 
917.82 


60.90 

1359.13 . 


48^90- 
^ 756.51"'' 


200.00 
0.00 


199.66 
34.43 


112.97 " 
2832.98 


84.67 

1962.52 


101.78 
2279.99 


79.81 

1652.54 


100 

. A , ...... 

nB::3r::-'" ::■^:;:: 


84.30 
1188.04 


70.84 
999.99 


51.21 . 
1039.26 


■ 43.98, 
681.22 


47.11 
596.34 


44.06" 
642.90 : 


110.54 
1751.84 


94.07 
1258.61 


66.81 
J18.70 


■ 57.02 
614.92 


,_ 59.28 
. 536.85 


53.61 ■ 
471.00 \ 


■ 100 

k 

25 


39.48 
- 365.17 


36.76 
341.61 


31.71 
203.34 


30.31 
156.66 


30.70 
- 191.41 


30.02; 

172.76 


36.85 
239.20 


36.51 
297.97 


" 35.16 
310.12 


3i;93 
171.47 


. 31.08 
140.91 


■ 30.98.. 
162, 65| 


100 


58.92 
^ 994.45 


54.05 
745.88 


39.62 
431.32 


36.75 

358.61 


35.52 
307.83 


35.71,. 
340.59 ■ 


25 


64.06'^ 
1192.78 


^^59.23 
953.26 


39^79' ■ 
407.40 


" 37.49 ■ 
324.22" 


\ 37707 7 
: 364.19 


34.87' 
315.87'-- 



^4^ 



! 



1 

1 
S 

i 



erJc 
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Table § 




Miins and Variances Whars e^-NCOsS^ ^^100) 

























Bias (x -8) 
n 


Bias (k -9) 
n 


Bias (x'-8) 


Bias (x -a)> 


So, of steps- 


Bias {x -a) 
■ n 


No/ of stips ; ' 




a 

_ n„ 


at 30 staps 


at 50 stipi 


it 100 stips 


vheti'stoppad 
wlth Rl "' 


when stopped 
with E' " 


vim stopped 
withR2 •„ 


when stepped^ 
with R2,, 

■ - — -- •■ - 1 






-.49 


.30 


.35 - 


-.88 ' 


' 20,33 


-.22 ' 


32.59 - ' \ , 


".[100„m42,0,0] 




160.93 


93.84 


48.73 


.23J.5r. 


1.11 


172.67 


222.70 . 


|H-« r . 

•.r ■t--i|i ! v.-v.^- ■ '. . ■ ,.' 




.30 


1 .10 


. .39 


-.03' 


23.82 . 

' f 


-:25 


33.11 ' T-,- 


f J, 




234.71 


148.69 . 


[ IT.n , 




80.88 


232,17- 


289,16 iJ-? 




■1 

n8' 


-.13 


.17 ■ 


.16 ; 


-.26 ■ 


■ 

■ '20.30 


-.01 


32.57 ■•• 




















27.18 




8.14 . ' ■ 


■ 39.86 


1.07 ' 


■ 29:24 


219,56 , ''I 


■■Vi'A;.:.:L:" ■ • - ..- 

- ■• ;. .- 


k 


.25 


.17 


.08 




25.43 ' 


.66 


" '30.56 ^ i 




— {c"4) 
fl 














^ ^ ^: 'r 


......! .... 




72.55 ^ 


^ 36.14 


19.10 


63.29 


176.93 




"203.33" : 

V I- 1 

1 i 
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■^===^^-:--- -=;-'- .. Footnotes. 

"^The original plan was to have 500 replications for each set of v - 
conditions* however, given the value of used tier e, a^ = £ converged 

slowly and for loma conditions the rule "stop If n « 200" was used 

for virtually every replication* We decided to use only 100 replications 

in these Instances 9 and those runs are noted In the cables* 

. 2_ ■ ■ ■ ■'■ " " ■■ ■ ■" ' ' ^ 

The "c" In this quote Is not defined In the ewe way as the "c" 

k 

^ g . - -- = . — . - 

In the definition of a = * 

■ / n ■ ■ n " ■ \. 



